+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20211002/B1i --output-run DEEP/20211002/B1i/science#step1/20240318T043624Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 14.4187 seconds; current memory usage: 0.178 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 1.2397 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 1.1912 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/pipeline_orig.qgraph DEEP/20211002/B1i/science#step1/20240318T043624Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.9322 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 20.0739 seconds; current memory usage: 0.376 Gibyte, delta: 0.198 Gibyte, peak delta: 0.185 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.376 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0261 seconds; current memory usage: 0.376 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.376 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20211002_B1i_science#step1_20240318T043624Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 1.4740 seconds; current memory usage: 0.376 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.376 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 2.2462 seconds; current memory usage: 0.378 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.378 Gibyte (main), 0.366 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 5ab70a62-52b5-43af-8369-20a29a3b56fa parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-18 04:37:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-18 04:37:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 04:37:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-18 04:37:40 proc_lsst.multi:170 [INFO] [multi] job_id 78108 proc_lsst.multi INFO: [multi] job_id 78108 2024-03-18 04:37:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:40 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 78108 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 78108 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 78108 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 1bbda8f47d16be3708fb57da9aff5551 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ac680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a3b444a9-6a91-4c75-a5ad-69ed1fcdf431_isr_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a3b444a9-6a91-4c75-a5ad-69ed1fcdf431_isr_1039672_36.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 8318f1dec18f7ad154bce9a092abe0a1 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8acf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/daf104bf-ba84-4a6c-a360-de03738ff476_isr_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/daf104bf-ba84-4a6c-a360-de03738ff476_isr_1039768_31.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 0d86af8e209f46bf855241f8b0b9a337 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f97b9ff7-9767-4e89-8011-ffec7ba1ab8a_isr_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f97b9ff7-9767-4e89-8011-ffec7ba1ab8a_isr_1039770_35.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 513fec5a5fb30c652fa7dd173be87718 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd2e1811-bac3-418b-807a-27882d5a838c_isr_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd2e1811-bac3-418b-807a-27882d5a838c_isr_1039672_50.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 86f26aa24ad24f70260d41b82a1693ed parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/493fbd14-5c1b-4bf5-a6b7-7b0981879060_isr_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/493fbd14-5c1b-4bf5-a6b7-7b0981879060_isr_1039768_14.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash ebeee5f886c70cf099ddd0c1974203b4 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8adee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ecc8c965-ea6a-4990-b0a2-3457d7c6d363_isr_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ecc8c965-ea6a-4990-b0a2-3457d7c6d363_isr_1039769_24.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash cd74cd1f96967008ad302fa84398d2bd parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1ec1c172-7e5f-45c1-a410-c9a00a015a12_isr_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1ec1c172-7e5f-45c1-a410-c9a00a015a12_isr_1039674_23.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 2ab158735373fa6d049757f4b1f23c36 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/644a1b94-d832-42ed-8afe-cdc09c8b60a8_isr_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/644a1b94-d832-42ed-8afe-cdc09c8b60a8_isr_1039768_55.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash ded64f79a0f7059af2f47326eb210d86 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c70fef74-952c-45aa-8432-a70e66311aaf_isr_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c70fef74-952c-45aa-8432-a70e66311aaf_isr_1039673_27.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash dc6439d73d3d6547de9a5f893ade0d13 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0d89c77c-f4d3-44b5-9a1f-19e8f6104a88_isr_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0d89c77c-f4d3-44b5-9a1f-19e8f6104a88_isr_1039770_20.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 4f1d3558bbde63c1488183b897bcd1b2 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64e33a9b-007e-4598-8ff1-bc49099f1361_isr_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64e33a9b-007e-4598-8ff1-bc49099f1361_isr_1039672_15.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash dad8f02032272a3e05ad632062f0f659 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a7f9c228-2da1-499d-a620-4fce8e9e932a_isr_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a7f9c228-2da1-499d-a620-4fce8e9e932a_isr_1039673_2.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash d7727d7cf4351310ebc4701ca1f08646 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d0811059-7b39-4c12-a485-7d6078099f7a_isr_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d0811059-7b39-4c12-a485-7d6078099f7a_isr_1039674_6.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash cca3d17a246bf32ab7b2b32c95d27550 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c76b0826-198a-4bf2-bfa6-bbceeca4587e_isr_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c76b0826-198a-4bf2-bfa6-bbceeca4587e_isr_1039768_32.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash ac6df29ccda08db67e104c43ed8e8c5d parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61b4bf9d-3f64-4250-a316-380d3cf135a9_isr_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61b4bf9d-3f64-4250-a316-380d3cf135a9_isr_1039768_53.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 344070072464d596f1191530810cb7e4 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8afce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/45dce31a-346d-4a8f-ac1d-2b0bb210a3ba_isr_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/45dce31a-346d-4a8f-ac1d-2b0bb210a3ba_isr_1039673_25.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 4c6b4bc249a4138ade4297aa3bab1462 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b7111778-4794-4823-980b-b35fd3791f0a_isr_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b7111778-4794-4823-980b-b35fd3791f0a_isr_1039674_37.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 9e2fe72fdf9c14101b5c4bf0a69ff4ed parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f2154792-5301-408c-acb3-24fc5b768bb6_isr_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f2154792-5301-408c-acb3-24fc5b768bb6_isr_1039674_21.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash feceb4d01392fb638d2f7a0b8116282b parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bef9535d-8989-47a6-96e7-f54cb8d232fc_isr_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bef9535d-8989-47a6-96e7-f54cb8d232fc_isr_1039769_34.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash fc1bd46ad69d2bf0861a86540ac0776d parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/eeeeda02-fe8e-44e7-9b49-0a55d72248f0_isr_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/eeeeda02-fe8e-44e7-9b49-0a55d72248f0_isr_1039769_36.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 751a17a75c6fd098e509c4c4d661e0f3 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/766af1f8-a676-417a-abe7-e3d578975c78_isr_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/766af1f8-a676-417a-abe7-e3d578975c78_isr_1039768_46.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 02077c11b45767be238d027da728263c parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e9f6fe89-8534-482e-824d-2a84949016c2_isr_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e9f6fe89-8534-482e-824d-2a84949016c2_isr_1039770_36.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c48d0ff84ecce9d583d5d5dc42e8494b parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/64b27773-9979-41d8-94cf-a9186b91a996_isr_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/64b27773-9979-41d8-94cf-a9186b91a996_isr_1039768_18.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash f47f359df5b176e8fda2ba2ed78b5713 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/554f5e58-dff4-4dea-95fe-3edd6a8e2909_isr_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/554f5e58-dff4-4dea-95fe-3edd6a8e2909_isr_1039768_2.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 248406f487454e947a602cd6dd503d4a parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/60369cd3-c064-4019-9217-5735c3202f90_isr_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/60369cd3-c064-4019-9217-5735c3202f90_isr_1039768_43.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 96bb82cccf1ef1f62a3b94b385de904d parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e11a28c7-0f3f-40b5-93a7-fb18611cf290_isr_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e11a28c7-0f3f-40b5-93a7-fb18611cf290_isr_1039674_22.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 831c6deb0f5053dc52e6aa801b7a18f1 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/17e7e44a-9b1e-4075-b307-f9ee8f2754ca_isr_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/17e7e44a-9b1e-4075-b307-f9ee8f2754ca_isr_1039768_8.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 07069c4c59345984efdbbc534c95a567 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e00914e3-7553-40d7-af1b-eb57d387c995_isr_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e00914e3-7553-40d7-af1b-eb57d387c995_isr_1039674_4.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash a92b78f4c2cf86a6466662818ac32ce0 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/853ba2a1-5218-4980-89c1-3057aedf7c69_isr_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/853ba2a1-5218-4980-89c1-3057aedf7c69_isr_1039768_42.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash d2016c2ba40fac7a6b3de080ca05116b parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/994e98c4-c041-41a3-900d-906293916b4c_isr_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/994e98c4-c041-41a3-900d-906293916b4c_isr_1039768_15.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash dade2794deee2925defadeb0e0c55dfb parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/46a1dd71-ada5-46dd-8f2e-cab53696e9a2_isr_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/46a1dd71-ada5-46dd-8f2e-cab53696e9a2_isr_1039673_7.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash e9a79f2fb6e7da2a4e3517deecbd9402 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5974d213-e1c5-4ea1-937c-6d59c2746b81_isr_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5974d213-e1c5-4ea1-937c-6d59c2746b81_isr_1039769_28.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash e41cb0287f12424c1dfe43cd10903601 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/578203ac-36df-4376-840f-bb5be383ccdd_isr_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/578203ac-36df-4376-840f-bb5be383ccdd_isr_1039672_27.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 3a6cf78ed8ed009787317b97e6c40b00 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/83cb17c2-9ea1-449c-94cd-be991290b3a4_isr_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/83cb17c2-9ea1-449c-94cd-be991290b3a4_isr_1039673_32.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 5f6258e7107abae07db4a8fc8abadd83 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c85f298f-1482-4f8a-adc2-346f466d1423_isr_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c85f298f-1482-4f8a-adc2-346f466d1423_isr_1039770_21.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 34f1401c54c008085e7033db006a63b7 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7babba24-191f-41bb-a684-ee26f89dfaad_isr_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7babba24-191f-41bb-a684-ee26f89dfaad_isr_1039770_10.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash f007d28be6d13fa404016f8f0dc19e8e parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54310d8e-7f5d-4b0d-bb35-8d5236fe603e_isr_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54310d8e-7f5d-4b0d-bb35-8d5236fe603e_isr_1039770_3.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 2580c7fe1c35d2938b44b2891971dae0 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/255d4340-7639-47bd-84ce-e4ac52c782d0_isr_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/255d4340-7639-47bd-84ce-e4ac52c782d0_isr_1039768_58.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash cfa4aaf4b21fdf1a3ed8a9bef2abf22b parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/73f5196f-4a9e-4902-a5a9-d9e49603f895_isr_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/73f5196f-4a9e-4902-a5a9-d9e49603f895_isr_1039674_12.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 49c49d1ed29fe66627b2cb58cd0ab111 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/156f9a6f-f7c1-4d45-a5fb-672b3f56069a_isr_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/156f9a6f-f7c1-4d45-a5fb-672b3f56069a_isr_1039770_58.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 91b47a54f6676d279befd1b02536b75e parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/1f7256f7-d2a1-4164-83b4-e1468a1055a9_isr_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/1f7256f7-d2a1-4164-83b4-e1468a1055a9_isr_1039672_13.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 478cdd4c9930286b2af9670ea30133e7 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/fa07acac-a19c-4927-90e4-23dbab897f5d_isr_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/fa07acac-a19c-4927-90e4-23dbab897f5d_isr_1039768_50.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 3359f6554f6ca5afa66fea6d084a29d9 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0fabdd96-12f5-43ad-9569-d6fe067a7369_isr_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0fabdd96-12f5-43ad-9569-d6fe067a7369_isr_1039673_22.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash b991c484daf27d6cc83bdf26d8292361 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7cc3f24e-88dc-4c51-a0f5-e565856a2e35_isr_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7cc3f24e-88dc-4c51-a0f5-e565856a2e35_isr_1039672_58.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 48a13e202ea0c04d04799dc11a9ba864 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d1a9adfc-88db-49d6-b46c-8ae5ffe761ff_isr_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d1a9adfc-88db-49d6-b46c-8ae5ffe761ff_isr_1039673_36.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash c25346798229f3b151c564c58e0f33af parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/06f1fa75-6f8e-4369-8b00-1fd154903d9c_isr_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/06f1fa75-6f8e-4369-8b00-1fd154903d9c_isr_1039770_33.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 5ce729593a644f904b0ae6805831db6f parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/df76f3a7-3859-40b2-9f02-d7516a9f931d_isr_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/df76f3a7-3859-40b2-9f02-d7516a9f931d_isr_1039672_40.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 1b5dc5d3a31c5f58fa9334592cffdf8c parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/24ea5d9c-e07d-497d-8e21-e867bf8bd4f8_isr_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/24ea5d9c-e07d-497d-8e21-e867bf8bd4f8_isr_1039768_62.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash a0591229b2c819611c83e4e7fbce0b40 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e6d71bd1-09db-4fae-93ca-d644a6764343_isr_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e6d71bd1-09db-4fae-93ca-d644a6764343_isr_1039674_40.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 21f474b4482c741141a9281e5ff35824 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61c2298f-9016-45dc-9002-206cccdff9f9_isr_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61c2298f-9016-45dc-9002-206cccdff9f9_isr_1039768_3.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 2bd89dc9235be354599f42651fb268b4 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d37b4d07-e081-4f1b-8312-36cdfd6d8fe8_isr_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d37b4d07-e081-4f1b-8312-36cdfd6d8fe8_isr_1039674_30.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 96fddbafe60444384c38e1a80115ee30 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f933dd07-1046-47af-900e-17d8a0670342_isr_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f933dd07-1046-47af-900e-17d8a0670342_isr_1039770_11.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 5d7616fa5068d69d70995a3698f8171e parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8e6b83f8-b368-44da-b9a7-56693f8a5204_isr_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8e6b83f8-b368-44da-b9a7-56693f8a5204_isr_1039768_24.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 11c184ed1a7dd298471915c23ae9b8cb parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/938e6cbf-c9b0-4091-889f-d0ab443f6c4e_isr_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/938e6cbf-c9b0-4091-889f-d0ab443f6c4e_isr_1039769_48.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 159a5fbbeba39849386e5409d73a0f3a parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c11b1c75-b7d6-458d-829b-05a01ed6546d_isr_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c11b1c75-b7d6-458d-829b-05a01ed6546d_isr_1039674_35.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 4f61cda82eb2ef95a0b1633b8f771b12 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/68261430-ccc3-4923-a72c-d6003d9bb7d6_isr_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/68261430-ccc3-4923-a72c-d6003d9bb7d6_isr_1039674_1.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash a029ea56282d126c21af5c8b38b9240f parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a0de1073-3699-47d1-a7ec-458ba7968a62_isr_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a0de1073-3699-47d1-a7ec-458ba7968a62_isr_1039672_39.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash c61978231f756f9ca30f09b48974d416 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9100e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/843b5d50-fae9-4f78-afbc-44cd046d4d33_isr_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/843b5d50-fae9-4f78-afbc-44cd046d4d33_isr_1039770_41.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 9c998ceac0a5b2b077643b0b6316322f parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8fbe1e8a-37af-472f-b369-93afd442ef88_isr_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8fbe1e8a-37af-472f-b369-93afd442ef88_isr_1039769_59.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 183abc6c0ea31b786b907a46a138c9cf parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/87a46004-8530-400a-9779-2d4da889b0ba_isr_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/87a46004-8530-400a-9779-2d4da889b0ba_isr_1039770_32.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 79964494b3bf2e79e86bf5dd014ccd13 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/66a7162a-fc14-44e6-a8c6-191cac61e9f1_isr_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/66a7162a-fc14-44e6-a8c6-191cac61e9f1_isr_1039673_6.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 606a2ebcb2d27441dabdaa54e524213f parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88f4c182-6fbc-4a33-9bc5-390d3c09a78b_isr_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88f4c182-6fbc-4a33-9bc5-390d3c09a78b_isr_1039674_48.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash db97e49255abb819d3dd5f9c23049568 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e0303cec-e056-41ec-b64b-09660376a9c0_isr_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e0303cec-e056-41ec-b64b-09660376a9c0_isr_1039673_17.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash e30ef5fae2aeca6c1a563a0567e8c77a parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9113a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/b2a23c03-0105-4f28-89be-cf7e58e547f3_isr_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/b2a23c03-0105-4f28-89be-cf7e58e547f3_isr_1039673_35.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash f173fccb38cae1380fef1f76ee4683fc parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9116c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1533ebea-7697-4043-ab14-c2fc22ff3746_isr_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1533ebea-7697-4043-ab14-c2fc22ff3746_isr_1039673_48.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash d96b1d38f1e06ebf8b6324c2b5fdb6a1 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9119e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/fb29da83-871a-43c5-a456-4b39916e1aff_isr_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/fb29da83-871a-43c5-a456-4b39916e1aff_isr_1039672_56.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 53142ca8ae75ceebac802193c9defcd0 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4500de69-44fa-4e13-9786-85d23bb970c1_isr_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4500de69-44fa-4e13-9786-85d23bb970c1_isr_1039768_44.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 9e7c440db18cd4ab325e9a06a431be67 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4d609123-234a-4ff4-b86a-fc2b519609b8_isr_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4d609123-234a-4ff4-b86a-fc2b519609b8_isr_1039768_16.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash edceedc3513fa5698ba2afb2fe57aa98 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/219a5b0f-4cc4-4b39-9b7e-0ec131e2279f_isr_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/219a5b0f-4cc4-4b39-9b7e-0ec131e2279f_isr_1039768_59.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 53adc3ca19c67f07c06e108394c2a3cf parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d5b37484-eb91-45a2-accb-cad3a0bd642a_isr_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d5b37484-eb91-45a2-accb-cad3a0bd642a_isr_1039673_14.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash b4041c5c3fb4eddede4c61026860245b parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/9cfe5630-2c9c-4030-a09c-49d082d877cb_isr_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/9cfe5630-2c9c-4030-a09c-49d082d877cb_isr_1039770_48.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 718fc2d456b860370845f34e93edc9af parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd78fe2d-2dee-49a0-aafc-75905dced438_isr_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd78fe2d-2dee-49a0-aafc-75905dced438_isr_1039674_36.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash ffe7f89902812612ed9b306cf50f28d4 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/44bb32a1-d38d-490d-82ea-36481bca8907_isr_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/44bb32a1-d38d-490d-82ea-36481bca8907_isr_1039672_2.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 05b5c437e88827c0711f1828fd871077 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9132e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e2955ecf-c11c-4941-ad73-2b616a359f40_isr_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e2955ecf-c11c-4941-ad73-2b616a359f40_isr_1039673_18.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 596a8dccad723d4e3a35dae6b99af6a2 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b4530209-2169-4810-9481-4f906967e8e0_isr_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b4530209-2169-4810-9481-4f906967e8e0_isr_1039672_17.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 3b469159776a289784966164a76c6e6b parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b289a6f3-f624-4560-be57-2fc80053fe6f_isr_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b289a6f3-f624-4560-be57-2fc80053fe6f_isr_1039769_43.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash ab558705867af04136b5ba3c61883498 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2fca352e-4058-4123-b056-adba8691987f_isr_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2fca352e-4058-4123-b056-adba8691987f_isr_1039770_29.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash b2ab67320f63126642095c4e5f472a15 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/efd21eab-a7e5-40dd-b0da-c186708f5280_isr_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/efd21eab-a7e5-40dd-b0da-c186708f5280_isr_1039672_16.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 2ea56403a6e6f98ed84f129e36b9b9f3 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9382c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/915df0c9-be1c-42ba-8c1a-5be8948223b3_isr_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/915df0c9-be1c-42ba-8c1a-5be8948223b3_isr_1039672_9.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash d992974e645a132bf93b6feccdedfc3a parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/35d92771-1fca-4fc9-b94a-b4ed56471356_isr_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/35d92771-1fca-4fc9-b94a-b4ed56471356_isr_1039674_47.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash d8fa6c63621988b107bf0eaf0caf3d22 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0eb1d2bb-589b-4ae5-a96d-0fe4721b9282_isr_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0eb1d2bb-589b-4ae5-a96d-0fe4721b9282_isr_1039769_39.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 6fed7d1f63c1b6a5988f812b30878560 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/48997116-3016-4ac2-8e20-9e1352e55b8d_isr_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/48997116-3016-4ac2-8e20-9e1352e55b8d_isr_1039768_61.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash caf34593190c5c801ff2e2be34668383 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/69a34123-57de-4015-bf92-2c6f6c8278ef_isr_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/69a34123-57de-4015-bf92-2c6f6c8278ef_isr_1039770_34.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 0198a9f42341ee9f0deb8ca8121c3c5c parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88294abe-b1de-4925-899c-f6fd62e1d86c_isr_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88294abe-b1de-4925-899c-f6fd62e1d86c_isr_1039674_10.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash c2391eca6f9b69c53ba2bdcf8f9d8ee4 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04df2adb-1006-4fd0-8623-5efa75494338_isr_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04df2adb-1006-4fd0-8623-5efa75494338_isr_1039768_13.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash ae0e5346ead7be3b42a0695a00788e0f parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/48e9fda6-54c9-461a-abf4-5d79737fd0da_isr_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/48e9fda6-54c9-461a-abf4-5d79737fd0da_isr_1039769_14.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 0b5e239966e6c90d5d39f5be5f6735fd parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ae202a23-0f7a-4b3e-9e4e-c78a4883ffa1_isr_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ae202a23-0f7a-4b3e-9e4e-c78a4883ffa1_isr_1039673_11.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash c3af703200145f3b9610ecb86d57e3df parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/cccf163c-09e4-43d2-ae03-c7a110cb82fd_isr_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/cccf163c-09e4-43d2-ae03-c7a110cb82fd_isr_1039769_46.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 552ded2a5773de002992f35fa829d30f parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d75208d5-b812-46a4-9475-5eb261eff473_isr_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d75208d5-b812-46a4-9475-5eb261eff473_isr_1039672_62.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 486f49c76cd5c16a4641cc6dcb8f6bae parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1219a93e-d33e-45bc-be25-1c12214bebf2_isr_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1219a93e-d33e-45bc-be25-1c12214bebf2_isr_1039770_30.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash fd2f6a30bed3c60c8f3ce8dc9c0e455c parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/366bc16f-26a2-41da-9afe-995ce82e830b_isr_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/366bc16f-26a2-41da-9afe-995ce82e830b_isr_1039672_52.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 6761811f4c80397193bb84b7eac781ad parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/2e80db22-f184-4f12-9ac3-23234b174952_isr_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/2e80db22-f184-4f12-9ac3-23234b174952_isr_1039672_23.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash da284928af5c997ec0622433800475ca parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/689c73ba-02eb-4175-ae34-749cfb92b638_isr_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/689c73ba-02eb-4175-ae34-749cfb92b638_isr_1039672_42.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 8322e5cbbd6805167109a85cd2d7c5be parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a5af4ba7-d5fd-4e70-8619-2f410cfd965c_isr_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a5af4ba7-d5fd-4e70-8619-2f410cfd965c_isr_1039674_33.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash f530e67128509b33a1e207072f20c799 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/306f8efc-0b19-4e46-8b36-32cbf41dafd0_isr_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/306f8efc-0b19-4e46-8b36-32cbf41dafd0_isr_1039672_8.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 4932a3532f5a1f84449181750eca2aba parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/d7bb4fed-c769-4206-b60c-3c6f93fdb9a6_isr_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/d7bb4fed-c769-4206-b60c-3c6f93fdb9a6_isr_1039769_29.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash f2dc10bc65f5c8e76bb40cae8cfb0960 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/781dd59c-34fb-438e-956b-e7f674183580_isr_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/781dd59c-34fb-438e-956b-e7f674183580_isr_1039769_30.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash bf2d6d1408795e2c94afcc87f808b64e parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b536da77-1afa-4655-aab0-3a141e4bc738_isr_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b536da77-1afa-4655-aab0-3a141e4bc738_isr_1039770_50.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash c32eb0e3d41b8455b443811f8fe44652 parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd251fc1-6fca-4cd5-9725-32107528b358_isr_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd251fc1-6fca-4cd5-9725-32107528b358_isr_1039672_26.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 1000b3ab8003933c0a656d6c98350279 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1bc8dfb5-0ed7-4cdc-9bab-7cb139a9d355_isr_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1bc8dfb5-0ed7-4cdc-9bab-7cb139a9d355_isr_1039674_25.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 428cc2721e5a1966ce67814b5c193b31 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1ed6b4e0-72a6-4088-8c96-95afe4dfbdfb_isr_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1ed6b4e0-72a6-4088-8c96-95afe4dfbdfb_isr_1039769_53.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash d3d21bf02ef897a73a0106ef883bf38d parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c75cad51-f644-4d28-9d69-1187de0cc0d6_isr_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c75cad51-f644-4d28-9d69-1187de0cc0d6_isr_1039674_58.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash cdfcdbf5f35858173f26c6dc48e0e0d5 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/429c9a8d-ac72-4ace-af59-202b3e455bcf_isr_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/429c9a8d-ac72-4ace-af59-202b3e455bcf_isr_1039769_55.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 45467c72cad86429b2b205daa75a5337 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/502eb62b-ab25-47df-815a-4e654a325b46_isr_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/502eb62b-ab25-47df-815a-4e654a325b46_isr_1039672_43.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 2bdcd1ba383b269531f1fd7085eec9ed parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/01c43af4-1438-4e2f-a278-4d5a4ea028e0_isr_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/01c43af4-1438-4e2f-a278-4d5a4ea028e0_isr_1039673_31.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 8e777381ef978a698fae778afd56fff0 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b2e38a1d-267f-43d8-b6bc-bc599573eb85_isr_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b2e38a1d-267f-43d8-b6bc-bc599573eb85_isr_1039672_22.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash f3b04a413a4f3798f0b7832f929575e4 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/dc591d2e-78a8-4253-8f4e-491bbf456fa9_isr_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/dc591d2e-78a8-4253-8f4e-491bbf456fa9_isr_1039770_42.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 4620c470ecc10b614be59581d9d36d85 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fb781d59-f291-40aa-a4b4-855b7ded74ec_isr_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fb781d59-f291-40aa-a4b4-855b7ded74ec_isr_1039769_44.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 969f995b9b7d45fba9b1a119c2a2a537 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b5912956-151e-4033-bd7f-0813b1a5d8b6_isr_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b5912956-151e-4033-bd7f-0813b1a5d8b6_isr_1039674_19.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 4be5b801cc579bbc78082d84d7c7883f parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a30528b8-2fee-4a19-82b6-2740e72e798d_isr_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a30528b8-2fee-4a19-82b6-2740e72e798d_isr_1039672_48.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 979d086ea9a036f0188e054c2f58201f parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a525def4-d556-4794-b147-d33714396db5_isr_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a525def4-d556-4794-b147-d33714396db5_isr_1039770_9.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 65159713246cdef5c82702b2f0bd014d parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/60a40a12-7025-452c-9c26-46cf6fa39108_isr_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/60a40a12-7025-452c-9c26-46cf6fa39108_isr_1039770_28.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 16b42f330d31da24bff54c1525d1d25c parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c8360159-eed8-4dd8-aa14-0dc94b91cb91_isr_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c8360159-eed8-4dd8-aa14-0dc94b91cb91_isr_1039768_20.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 3be1a71ff794ac80d47b8db3b9f772bd parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b8663665-6a24-4cef-9d2a-c86373e9eade_isr_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b8663665-6a24-4cef-9d2a-c86373e9eade_isr_1039768_29.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 2ee3c3f023de38c32c10f8bf04243a5c parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8e14eabb-e403-4d7d-bdb8-30774b38d67c_isr_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8e14eabb-e403-4d7d-bdb8-30774b38d67c_isr_1039769_15.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash c4ebc44a3f65d55595b05a3cfd2c2910 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ba9b878e-5bdb-44f7-b67d-2f3e1b2ddaa1_isr_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ba9b878e-5bdb-44f7-b67d-2f3e1b2ddaa1_isr_1039673_12.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 4f9bb90145675890312ee78c30166d87 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/59156357-8474-400b-b0c0-345dd98b2a1c_isr_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/59156357-8474-400b-b0c0-345dd98b2a1c_isr_1039770_6.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 23a019b719a2e9e5c31248f8d4023358 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0a428ed0-77ac-4825-8581-ed0771abb165_isr_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0a428ed0-77ac-4825-8581-ed0771abb165_isr_1039673_43.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash dbd61c3b3203e43a5d3965c72a5d066c parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a73c514b-6c4e-4703-9a5d-ba41be403a5d_isr_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a73c514b-6c4e-4703-9a5d-ba41be403a5d_isr_1039770_46.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash c921b4648e485ad0b5f7ed3798766d92 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1951b5ed-cf68-4f99-a9bd-16e83c2265f0_isr_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1951b5ed-cf68-4f99-a9bd-16e83c2265f0_isr_1039673_33.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 27ca97a11d1309f5d2131570fdf133ff parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2019cf48-ca29-49aa-b22a-78d917218d69_isr_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2019cf48-ca29-49aa-b22a-78d917218d69_isr_1039770_56.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash db228d7ed1b867014e46f994dd6f8af8 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8712a2ae-27fd-4520-b3bd-b9eea3b4f1a6_isr_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8712a2ae-27fd-4520-b3bd-b9eea3b4f1a6_isr_1039674_51.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 064e8120a2dabfc571328b1f71462789 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:41 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/67114727-7d7d-4bb2-9649-2696703bfd24_isr_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/67114727-7d7d-4bb2-9649-2696703bfd24_isr_1039674_57.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 5924b507c809b8efe9ffa842fcd7491a parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d500ff7-0815-4968-9048-51cd60b83785_isr_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d500ff7-0815-4968-9048-51cd60b83785_isr_1039673_5.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash 2d7229e00df6b425e89b2e6f1754ccb3 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a1641c1b-68e2-42b6-b941-c0364d475cfd_isr_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a1641c1b-68e2-42b6-b941-c0364d475cfd_isr_1039673_8.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash df5b6ad455531f74a5ee8e3b53d7468b parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3c46d659-7384-4cb2-8a24-5ad0d8805d39_isr_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3c46d659-7384-4cb2-8a24-5ad0d8805d39_isr_1039673_30.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 362ba3ad71f23e97ca63f380e4732f67 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3e47d515-0bbf-42a1-b734-0f35fd7abdce_isr_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3e47d515-0bbf-42a1-b734-0f35fd7abdce_isr_1039769_32.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash d0842b2d58fc7679bd038a29e4279a51 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f22eb894-5dce-48a2-9ba7-a20e72a05da2_isr_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f22eb894-5dce-48a2-9ba7-a20e72a05da2_isr_1039769_62.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 1ef2f33ff9aea03102d311e284adf44f parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7cc318e3-25ca-47fb-8150-ba76cd64f2ff_isr_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7cc318e3-25ca-47fb-8150-ba76cd64f2ff_isr_1039673_46.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash fadaeee92e21d777df0022991c732c9e parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 129 parsl.dataflow.strategy DEBUG: Executor multi has 129 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-18 04:37:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-18 04:37:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 04:37:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4512c4da-d9db-49ce-9b99-f94cef217c7c_isr_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4512c4da-d9db-49ce-9b99-f94cef217c7c_isr_1039674_26.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash f9da059e62762928316e05f1793f1c9c parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/95d5cb3e-256c-4931-898f-1bb20c659156_isr_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/95d5cb3e-256c-4931-898f-1bb20c659156_isr_1039770_57.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash e5aa0e03300468dc82a082a2314b1c7e parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/57b22b56-3f3d-483a-a3d4-0d05ee67a11e_isr_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/57b22b56-3f3d-483a-a3d4-0d05ee67a11e_isr_1039768_33.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash f4416ff466e3fa7a5481aacee246b09d parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64a3c2a5-99a7-4d72-b2a8-2584dbd5f762_isr_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64a3c2a5-99a7-4d72-b2a8-2584dbd5f762_isr_1039672_18.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 19e9eb3ea984e0432634d36afa76fed1 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/253c6072-4d13-4841-b8f5-d911ef28e904_isr_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/253c6072-4d13-4841-b8f5-d911ef28e904_isr_1039769_41.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash a22d2a169eb3693a704ca2a2757c6cb6 parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3df7d690-1e64-42e6-aaf1-6da71af4c3e3_isr_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3df7d690-1e64-42e6-aaf1-6da71af4c3e3_isr_1039673_20.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 63b0edd5ad252ae7c1a91650ab3adda0 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1a970642-948c-4634-8c9f-2ca549e87987_isr_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1a970642-948c-4634-8c9f-2ca549e87987_isr_1039769_21.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash aa5e39136bf40842c7ffa9a3e6e3f131 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d461f5f4-a7ae-4f3c-8643-0a5bd23b391a_isr_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d461f5f4-a7ae-4f3c-8643-0a5bd23b391a_isr_1039768_26.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 867300780258723a0ddb4ab1f5ab64de parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1daff6da-2e2e-48e7-95f6-bbbfec85dbf0_isr_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1daff6da-2e2e-48e7-95f6-bbbfec85dbf0_isr_1039769_57.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 08ad7b500f226ece234d0d5a41e16600 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b3e11ec9-b539-47cf-af52-c4c1b908fc8e_isr_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b3e11ec9-b539-47cf-af52-c4c1b908fc8e_isr_1039768_48.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash e860d74e054c34d0e3541c78015bd36e parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c241f33e-3c4d-4ada-9c7e-ba108abaa436_isr_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c241f33e-3c4d-4ada-9c7e-ba108abaa436_isr_1039770_37.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash e490d38f09efebbbf3803501dedffe53 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/144a5c82-48b1-4557-be48-511347e8a07a_isr_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/144a5c82-48b1-4557-be48-511347e8a07a_isr_1039673_45.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 73f25e6755b21e3a03cf7114d58d30d3 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/927b506b-0af5-4ce0-b59e-9a4eb53bd417_isr_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/927b506b-0af5-4ce0-b59e-9a4eb53bd417_isr_1039769_10.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 12b94c55db4ec5e204453ae563a84bff parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9454bbc1-f149-4bfb-a22a-b6de74b2cb40_isr_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9454bbc1-f149-4bfb-a22a-b6de74b2cb40_isr_1039768_40.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash ad1a46f0bcd7b44a8e53856296c4bb77 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1e607320-51de-4af5-a9c9-bf0d9a5cb390_isr_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1e607320-51de-4af5-a9c9-bf0d9a5cb390_isr_1039674_62.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash f55a880acb2468a7cd3eb2e8404a7274 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3d665c1-a973-4fee-91fe-75777420339e_isr_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3d665c1-a973-4fee-91fe-75777420339e_isr_1039673_4.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash c5df2399602351e9cafe088e640172da parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/85cfbba5-269f-40e5-aaa6-4c6d17ba0402_isr_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/85cfbba5-269f-40e5-aaa6-4c6d17ba0402_isr_1039769_27.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash ef6d1a4369d12d0daaa6cb6f83fb79c2 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bb8017ce-1432-4169-bce4-166d5b2cffd5_isr_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bb8017ce-1432-4169-bce4-166d5b2cffd5_isr_1039769_33.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash a96835c18c8dd6901c49a6ecd6a0d2d0 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/50d19a15-b33b-452d-935a-3be7a6d1b1d2_isr_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/50d19a15-b33b-452d-935a-3be7a6d1b1d2_isr_1039770_24.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash a78a99806b8f5932157c8e79aaf1d77b parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aef2db73-c7a4-4d7a-803c-a857aea47f55_isr_1039674_50.stdout 2024-03-18 04:37:42 proc_lsst.multi:170 [INFO] [multi] job_id 17193637 proc_lsst.multi INFO: [multi] job_id 17193637 2024-03-18 04:37:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193637 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193637 parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aef2db73-c7a4-4d7a-803c-a857aea47f55_isr_1039674_50.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17193637 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-18 04:37:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 9be987897fbe7e2e11b73812d7eb874f parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3822a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/40a25c72-392c-4a5e-acc6-54406aa1a053_isr_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/40a25c72-392c-4a5e-acc6-54406aa1a053_isr_1039674_55.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 14e582b158f286ea168fe44ffd3e60a7 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b16192ca-b849-463a-849c-a91caa94e163_isr_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b16192ca-b849-463a-849c-a91caa94e163_isr_1039770_47.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 39e12091d986b8af2cd5494e4fdfaf0b parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/048c9b7d-a012-420d-91b8-1fec876062f8_isr_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/048c9b7d-a012-420d-91b8-1fec876062f8_isr_1039770_60.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash afc8d0526afbea45f20b5e226e6ddf15 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4dea901f-bd07-4612-9eba-defd265a6abe_isr_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4dea901f-bd07-4612-9eba-defd265a6abe_isr_1039673_16.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash c764d7cbe2212e1512f61568bd368bd3 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3832e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a80acb93-f353-4af1-84a4-0e5d551313b2_isr_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a80acb93-f353-4af1-84a4-0e5d551313b2_isr_1039770_15.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash c9c2c3f66515ca357809e534b019f338 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d52ec15-95fd-463f-b459-c96e53d47963_isr_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d52ec15-95fd-463f-b459-c96e53d47963_isr_1039673_58.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 87981780ae4a5eb6b34b308cbf4dc64f parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b29e3e8a-5451-4046-a326-07e4d0716113_isr_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b29e3e8a-5451-4046-a326-07e4d0716113_isr_1039768_28.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 9b3a0453ebd03908a93e06dcdd5e4303 parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a72fb2f-524e-4f91-8db3-ddf2419306e6_isr_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a72fb2f-524e-4f91-8db3-ddf2419306e6_isr_1039674_16.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 3e65a92b0e488568832c23f3715ad8f4 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3ed58890-7333-4dbe-94e9-cee946119578_isr_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3ed58890-7333-4dbe-94e9-cee946119578_isr_1039770_59.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash 11ce4164b3f75db1363d1785580343da parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fd02e8bf-bb34-438c-a8b8-79fdf09066db_isr_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fd02e8bf-bb34-438c-a8b8-79fdf09066db_isr_1039770_55.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 0cec42fc0141a154de1c948fed2b269d parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f5903255-b522-4741-b516-3d60e8a2a143_isr_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f5903255-b522-4741-b516-3d60e8a2a143_isr_1039673_34.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 6629a34e50472512312971bec9201312 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5a0a7c0d-39b8-4024-a41d-5fc95384ba56_isr_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5a0a7c0d-39b8-4024-a41d-5fc95384ba56_isr_1039673_26.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash a007d2bebcbbf794a103298122ed2adf parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7de3940d-c6bf-4e98-8cee-65a39e3c1eb1_isr_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7de3940d-c6bf-4e98-8cee-65a39e3c1eb1_isr_1039769_18.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash ae72f3efb212ec649426e00571cbe9f2 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/6aecd294-7b09-4110-aa08-430393361b09_isr_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/6aecd294-7b09-4110-aa08-430393361b09_isr_1039673_52.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash 4256c4bf5138299faeb8158448b1606d parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/219fc9e7-9286-4931-bc74-301956d13214_isr_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/219fc9e7-9286-4931-bc74-301956d13214_isr_1039673_42.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 34a2d07d1e3ef45dda15431e8d7cced4 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:42 proc_lsst.multi:170 [INFO] [multi] job_id 17193638 proc_lsst.multi INFO: [multi] job_id 17193638 2024-03-18 04:37:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193638 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193638 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17193638 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-18 04:37:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f809532-e050-45b1-965a-b3e78bef0e28_isr_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f809532-e050-45b1-965a-b3e78bef0e28_isr_1039768_6.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 556e1d432399448b76bc530318353a15 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6923542c-ba54-4bec-a5b6-9749750c8c47_isr_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6923542c-ba54-4bec-a5b6-9749750c8c47_isr_1039768_47.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 9a04e03a8871cebb92f9a0727259dbc8 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3c8b161-0ea4-4604-a28b-5b3a322c0365_isr_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3c8b161-0ea4-4604-a28b-5b3a322c0365_isr_1039673_53.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash e84a15520f58f3e28a3afbf0439e9959 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/34e8d805-5a4c-4263-8c50-6cd7f6a500a0_isr_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/34e8d805-5a4c-4263-8c50-6cd7f6a500a0_isr_1039674_17.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash cf4d4afbf49e1fb21d28591657877736 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f31173e9-98f6-40ee-95e1-b976c204f41b_isr_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f31173e9-98f6-40ee-95e1-b976c204f41b_isr_1039674_46.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 81d3d6ea450697426bcdcdcd5afb4db9 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b4cd7daa-f1bd-4737-94de-c63546f9c1eb_isr_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b4cd7daa-f1bd-4737-94de-c63546f9c1eb_isr_1039769_5.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 3e8f3f7736c0c1bb1692d9338a92a230 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/62a05f4e-8dc3-41f9-997f-bec15eb5b3f1_isr_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/62a05f4e-8dc3-41f9-997f-bec15eb5b3f1_isr_1039674_43.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 71416e5771461bfdb98d526c07525bf1 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1248a035-ddae-4abc-b020-f7f3d7eaedee_isr_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1248a035-ddae-4abc-b020-f7f3d7eaedee_isr_1039768_57.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 3e4aa2fa6df61927e61247bb04905350 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8551df42-f3eb-42a8-a95c-6ac09df51e80_isr_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8551df42-f3eb-42a8-a95c-6ac09df51e80_isr_1039674_54.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash bc73c46042789e6e9aba527b27eaec86 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3baf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/cb562569-fa7d-4593-bd77-334041b9e71f_isr_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/cb562569-fa7d-4593-bd77-334041b9e71f_isr_1039673_57.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 5d06653b0daed0a8cc6037fd7dedd1a3 parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/472eb18f-23b8-461e-8c92-a059c59d2aed_isr_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/472eb18f-23b8-461e-8c92-a059c59d2aed_isr_1039673_15.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash 71268a912f102563ef5bc3da8e7a248d parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/612e4672-4a17-4141-9e55-50ab9bc331d4_isr_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/612e4672-4a17-4141-9e55-50ab9bc331d4_isr_1039769_13.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash f1a350fdf826a56f245b7ed325745233 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/231150fe-ea6b-43ab-8d72-127dffbf0e9b_isr_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/231150fe-ea6b-43ab-8d72-127dffbf0e9b_isr_1039770_25.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash 9e256f22aad5938852cf7d08596cdd8f parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5a61698-960c-48cf-a0b1-cc85f88e5d48_isr_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5a61698-960c-48cf-a0b1-cc85f88e5d48_isr_1039769_1.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash dca1802c690859a37fac06e009fe4e20 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/a2fe2589-5603-42f8-8df4-5497ed3b4c19_isr_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/a2fe2589-5603-42f8-8df4-5497ed3b4c19_isr_1039769_54.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 07b3e914d1fa706c3c40ab6c758375bf parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/29eca76e-96ca-447d-bd25-5326933b5813_isr_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/29eca76e-96ca-447d-bd25-5326933b5813_isr_1039768_5.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 007a700f95057b1a627833bfa1e87d09 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/47808565-ca17-4ac5-b5c5-fec836c0ddcc_isr_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/47808565-ca17-4ac5-b5c5-fec836c0ddcc_isr_1039674_53.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 4f88ab049c5655777090ca1dff942e7b parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/5c69f005-3b6d-4d36-89ba-7a38134fca42_isr_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/5c69f005-3b6d-4d36-89ba-7a38134fca42_isr_1039674_61.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 9205a2a39ccf26a17042dcd45d5e7073 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea489ef2-40f4-43be-b146-385a6b00d5e6_isr_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea489ef2-40f4-43be-b146-385a6b00d5e6_isr_1039769_47.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 1c320bcb7dc7d151c4db904df6811f6d parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/25a0fcec-2512-4d67-9190-5ca47646c8d1_isr_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/25a0fcec-2512-4d67-9190-5ca47646c8d1_isr_1039770_12.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 82d9ea4c2a9cd148acb0e1177d368972 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/3dc521d9-5091-4b8c-b943-050334525fa3_isr_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/3dc521d9-5091-4b8c-b943-050334525fa3_isr_1039674_13.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 69d966e094093e3ae4ba20728d865bc3 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/8096b4b2-734b-40b0-81c9-456ae2eaa67a_isr_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/8096b4b2-734b-40b0-81c9-456ae2eaa67a_isr_1039673_10.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash c0e849b5701470917ddab90cd301f4bb parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/10caed27-4d66-4826-bb02-51ea92095ae9_isr_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/10caed27-4d66-4826-bb02-51ea92095ae9_isr_1039674_5.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash ad5f5dd65b47d4e0c6a0de949eb151cf parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/42d0dd8b-bdca-44b7-9d98-50f0993d3657_isr_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/42d0dd8b-bdca-44b7-9d98-50f0993d3657_isr_1039672_12.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 960fbbe3d16cebfec0c04c49e29870e9 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd3590f6-d425-4cd3-aec5-eb1f63cb1d49_isr_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd3590f6-d425-4cd3-aec5-eb1f63cb1d49_isr_1039674_52.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 571fbcec90aea84d5435d748219c21fb parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b49e8b96-94e7-4563-be48-0049e7c098dd_isr_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b49e8b96-94e7-4563-be48-0049e7c098dd_isr_1039672_1.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 1029b13f004d1993d83c6ac3363965f9 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ebdf5f65-aea1-4d23-a4a5-abd0e280fad0_isr_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ebdf5f65-aea1-4d23-a4a5-abd0e280fad0_isr_1039673_61.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash c53f6679eaa733f46a93115aabde24d5 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/31025ffc-d2b0-45b9-ad3a-d39f35127e51_isr_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/31025ffc-d2b0-45b9-ad3a-d39f35127e51_isr_1039672_54.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash a88bec5d30e778d83e7b35b9c59b3753 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/e0723edd-3904-4a7f-a461-6853643f37f5_isr_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/e0723edd-3904-4a7f-a461-6853643f37f5_isr_1039672_21.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash de3ad4ed56ecaa3950a9be383d665724 parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/277f1ef7-d21b-4b6d-b0a1-09effb22c0b9_isr_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/277f1ef7-d21b-4b6d-b0a1-09effb22c0b9_isr_1039769_60.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 5b93a2eddb45671f2a8e82393195779c parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/49c44b1e-8cb6-4bcf-943f-a9affcaa85a3_isr_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/49c44b1e-8cb6-4bcf-943f-a9affcaa85a3_isr_1039672_34.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 53f093cb047cceb1c191fc799a7d60a2 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/0806d547-b491-45be-9893-0cb17c82463f_isr_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/0806d547-b491-45be-9893-0cb17c82463f_isr_1039768_56.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 6fc582eaf7d6d7443fc6099f0ad7c77a parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7f16bf58-2be1-4f0b-b481-b024d2e5e3bb_isr_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7f16bf58-2be1-4f0b-b481-b024d2e5e3bb_isr_1039770_16.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash b68645d6a96d2dfd92c56395050e4769 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/96a9f206-3717-44ed-a79f-ffc388a39ca8_isr_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/96a9f206-3717-44ed-a79f-ffc388a39ca8_isr_1039674_24.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 41a758426acc50337c6d093585401887 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ff8dc757-22ed-4af5-9be3-ffd27b4d5c4f_isr_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ff8dc757-22ed-4af5-9be3-ffd27b4d5c4f_isr_1039770_54.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash e876576eedf78f81970558d0c2f971d2 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dd9dac18-1ca9-4fd0-975b-13b1eb7d4584_isr_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dd9dac18-1ca9-4fd0-975b-13b1eb7d4584_isr_1039769_42.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 6a528fd2ca5161dc91b6a9c906c75fb4 parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:42 proc_lsst.multi:170 [INFO] [multi] job_id 17193639 proc_lsst.multi INFO: [multi] job_id 17193639 2024-03-18 04:37:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193639 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193639 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17193639 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-18 04:37:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/71a322a9-869f-42cc-9e83-481299d8edc2_isr_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/71a322a9-869f-42cc-9e83-481299d8edc2_isr_1039770_38.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash d7403f35239a8019f57bcf6efa2b2764 parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/91411994-0b8f-4c66-b1de-a3180a2510eb_isr_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/91411994-0b8f-4c66-b1de-a3180a2510eb_isr_1039769_56.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 711286ec9be1135b09fa5b8d94750894 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/db16e0c6-d593-4fe7-9f23-c09418d554dc_isr_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/db16e0c6-d593-4fe7-9f23-c09418d554dc_isr_1039672_55.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 2449b0beac48912b3b94a934818fdfc7 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4471920d-a424-4590-97d9-3993bf080fbc_isr_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4471920d-a424-4590-97d9-3993bf080fbc_isr_1039674_34.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 7ef110f161cdf677dff26675fcef50f5 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c076dacf-a42f-4627-a753-6fa4d62212e6_isr_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c076dacf-a42f-4627-a753-6fa4d62212e6_isr_1039768_30.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 96178167632ca9d23b9a60bdffc760ff parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a4463dee-a82d-406d-bb06-8606ec21b0cc_isr_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a4463dee-a82d-406d-bb06-8606ec21b0cc_isr_1039673_59.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash 6ef171f1116a39ddf49365a953686baa parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/903319e1-0242-40bc-bee6-8611fc3ac66a_isr_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/903319e1-0242-40bc-bee6-8611fc3ac66a_isr_1039768_7.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 9517171ab9d85da1d3d819710ffc6564 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/12d65d0b-7450-428d-b879-0182b7100009_isr_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/12d65d0b-7450-428d-b879-0182b7100009_isr_1039769_52.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 2f6e68ff110664d3336f1f97b7ebc28a parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5d873564-854d-4d81-a64d-9b35baeb805a_isr_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5d873564-854d-4d81-a64d-9b35baeb805a_isr_1039673_37.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 7a31014b19a790067c22d7576fb25605 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e4a4188c-8dc2-4782-8a60-ea86fc27c832_isr_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e4a4188c-8dc2-4782-8a60-ea86fc27c832_isr_1039768_36.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash a0fafe07f6056a9be716dc59be067a8a parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/41fc38b5-9d33-4bd5-bd10-4f6d1fcf8dc4_isr_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/41fc38b5-9d33-4bd5-bd10-4f6d1fcf8dc4_isr_1039768_34.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 7d31733971106d3b2fe11061ccbec1bf parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/49da0891-3fab-4b73-87b0-a9176a9d28d9_isr_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/49da0891-3fab-4b73-87b0-a9176a9d28d9_isr_1039768_23.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 01b908caad899f07db3dfa26849dd8d9 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/906cb417-a74d-4475-85c9-270373ae78ba_isr_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/906cb417-a74d-4475-85c9-270373ae78ba_isr_1039768_17.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 21682182283a765dfccc07656e20e357 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/60badfaf-c676-4c41-9c42-26db9f3de269_isr_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/60badfaf-c676-4c41-9c42-26db9f3de269_isr_1039672_44.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 0eda5fc73f50ec36925f682f851538dd parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59b4980-9e30-4beb-ab3e-3314b9ed8d07_isr_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59b4980-9e30-4beb-ab3e-3314b9ed8d07_isr_1039672_59.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash c08476dc303839bc162e90f616e85101 parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8b2cc1b3-8222-4d38-b961-9a81a89bb2c0_isr_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8b2cc1b3-8222-4d38-b961-9a81a89bb2c0_isr_1039769_35.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 7801e3e109516c0d9e11d45b4009e2a5 parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/be4ecd6e-de8a-486f-9cd6-81cfd575c905_isr_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/be4ecd6e-de8a-486f-9cd6-81cfd575c905_isr_1039673_44.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 8277974bec75bbc6b82d0e32b4d94d29 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a31d5ced-9faf-46e6-b720-6e4110fa6555_isr_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a31d5ced-9faf-46e6-b720-6e4110fa6555_isr_1039674_27.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash ae58bffd32a4050c6160baddf9db1d41 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3262cd85-d5c8-4d6e-8da1-6bbe17df7135_isr_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3262cd85-d5c8-4d6e-8da1-6bbe17df7135_isr_1039769_45.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash eee9960a2283647d9b7d740baf68c747 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2780e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a9d41983-a25b-4b96-b0fe-6ff710399fde_isr_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a9d41983-a25b-4b96-b0fe-6ff710399fde_isr_1039672_10.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 879b5a73a278ed6c62c9828943395a40 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:43 proc_lsst.multi:170 [INFO] [multi] job_id 17193640 proc_lsst.multi INFO: [multi] job_id 17193640 2024-03-18 04:37:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193640 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193640 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17193640 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-18 04:37:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/787a2710-880e-4f40-a3f2-553da330ea9d_isr_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/787a2710-880e-4f40-a3f2-553da330ea9d_isr_1039768_60.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 0fedb1fdc5858d743600861fc6e1ca8d parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2784a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d30de913-0188-4c08-bb68-11f86f6e744e_isr_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d30de913-0188-4c08-bb68-11f86f6e744e_isr_1039673_9.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 2c3734540d0bdc97e6cc8dac96d1d57a parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ff74e4d1-b2d3-47eb-8809-f35098480533_isr_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ff74e4d1-b2d3-47eb-8809-f35098480533_isr_1039769_61.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash e8c809bb5eb841212d2b49bbe922a21d parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c9ea411a-5a65-47f9-a5ef-1323e6bc74d1_isr_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c9ea411a-5a65-47f9-a5ef-1323e6bc74d1_isr_1039674_60.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash 60b40379e7566bdaf5f693cff21cccb6 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/91cc2541-7795-446e-826c-63a5287a113a_isr_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/91cc2541-7795-446e-826c-63a5287a113a_isr_1039672_35.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 378557d1475d2cf9bf3dabfbe71f3849 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2791c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/ca6aed33-352d-430a-a211-5c41e798fc4c_isr_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/ca6aed33-352d-430a-a211-5c41e798fc4c_isr_1039674_20.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 613591ce2cda65bcdaa33f401dbf71bb parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/5918e687-f605-4d77-a267-7369c56ee10a_isr_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/5918e687-f605-4d77-a267-7369c56ee10a_isr_1039768_49.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash 985a2b4754d53596fa02e64c5f677e49 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f82265ea-5ca6-4a4b-8dad-91a751073a44_isr_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f82265ea-5ca6-4a4b-8dad-91a751073a44_isr_1039768_51.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 557ae374a49047a430e30a52b7d7ee5c parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/bf0dd6cb-a17c-4698-b36a-a6e9adfb5fea_isr_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/bf0dd6cb-a17c-4698-b36a-a6e9adfb5fea_isr_1039673_29.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 2f55d97f93320cf74dc6d4bf985dce39 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/644833b5-4463-474a-be28-0f38abf3b278_isr_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/644833b5-4463-474a-be28-0f38abf3b278_isr_1039674_28.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash e3bd777e3ee2bea868f38811ff5dc2e4 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2799e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/557aa162-a6c3-4ab9-bf54-760744381707_isr_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/557aa162-a6c3-4ab9-bf54-760744381707_isr_1039674_8.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash 80baa260ffd1c5f356282ecdf5954007 parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/83d5ea87-6d5f-443f-a9c6-abb6b2080ace_isr_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/83d5ea87-6d5f-443f-a9c6-abb6b2080ace_isr_1039770_4.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 591fb99b20a230b54b8708c9f8bda4b5 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9c0866af-5e47-4d16-bc21-f9eb9274b415_isr_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9c0866af-5e47-4d16-bc21-f9eb9274b415_isr_1039673_24.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash b556f3a26326629902af20a68262d3cb parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f00ce571-cd13-4753-957e-0e3d96f4da22_isr_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f00ce571-cd13-4753-957e-0e3d96f4da22_isr_1039770_52.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 0848e3310bf32a29e36b5421138a1e9f parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c8a8db4b-2e95-4717-aa56-8837b63ecc0f_isr_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c8a8db4b-2e95-4717-aa56-8837b63ecc0f_isr_1039674_9.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash 610cca6e1a5421c5d31784bc8004df22 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/662902c8-5f0f-40bd-bbb2-a5c6411ebef8_isr_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/662902c8-5f0f-40bd-bbb2-a5c6411ebef8_isr_1039673_49.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash d499e0447346e7609cda2563aa9d93df parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a9dc094-7d86-4b5d-a1b8-7e064f60020d_isr_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a9dc094-7d86-4b5d-a1b8-7e064f60020d_isr_1039674_49.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 41738c39e170c2d546fed454d8fd0649 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f050114f-9a20-4038-84d8-5d77ecefeb07_isr_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f050114f-9a20-4038-84d8-5d77ecefeb07_isr_1039769_8.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 1dfbd44b9eaeb5d3d6da9ea44f91d9a3 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ac4e3a50-5e26-4f2b-b735-b9eac6ab5b83_isr_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ac4e3a50-5e26-4f2b-b735-b9eac6ab5b83_isr_1039672_46.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash f261bd186147fe57734dc7e8170f7aab parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c770f2d5-d935-44fa-b15f-5d495c9a5d1a_isr_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c770f2d5-d935-44fa-b15f-5d495c9a5d1a_isr_1039769_49.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash de991897b55f6772688a9431eb0a99b6 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/4e8578f0-3cea-4d7d-bd31-c1a9ab22b921_isr_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/4e8578f0-3cea-4d7d-bd31-c1a9ab22b921_isr_1039770_27.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 3e6a3c4e91b36b4eae10001a84117977 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fbe04133-9695-4edd-9286-2fc66d8695fb_isr_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fbe04133-9695-4edd-9286-2fc66d8695fb_isr_1039769_58.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash ceccc9735db3246d7b5cdbd7a03d918f parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/269494ee-c109-4f39-ace9-e1d233a58351_isr_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/269494ee-c109-4f39-ace9-e1d233a58351_isr_1039672_19.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash e6a584e53cd0fc2228265d420478c4cd parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59928f6-cf80-4041-b15a-e5d99c5ca2e7_isr_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59928f6-cf80-4041-b15a-e5d99c5ca2e7_isr_1039672_20.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 69a26dc2fe24f705892fe29a38c19afe parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f95d761c-ff47-40c2-8c1a-ab4a97cf8ed6_isr_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f95d761c-ff47-40c2-8c1a-ab4a97cf8ed6_isr_1039769_16.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash a4cb61fe00957a1e99aa0bef83c78fc6 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f43f747-b97e-4b4d-9249-3fb0cef312b6_isr_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f43f747-b97e-4b4d-9249-3fb0cef312b6_isr_1039768_4.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 45f4a40a3a3cd15b984b7c5855387333 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ce6ee819-0dbf-4d2c-8200-b1aa8176039e_isr_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ce6ee819-0dbf-4d2c-8200-b1aa8176039e_isr_1039673_54.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 5fa45152d38d5bfae48ecb0511d99dc9 parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5e3d7eb3-7007-4ade-9d25-1ba6b42d4e32_isr_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5e3d7eb3-7007-4ade-9d25-1ba6b42d4e32_isr_1039769_9.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash bd612636db242caefd595364051a219c parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ca4b8df1-b095-4459-afd8-51afd15c723f_isr_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ca4b8df1-b095-4459-afd8-51afd15c723f_isr_1039770_2.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 15105021b8a7379292ac178af5bb6d31 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7f25bd96-3e09-4cc3-a7fd-f21651e09f0e_isr_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7f25bd96-3e09-4cc3-a7fd-f21651e09f0e_isr_1039769_6.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash ca18f3e8177e73bcc0ac5cbea64106e2 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/dce2b500-599b-4ffc-a35b-2cd415642d33_isr_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/dce2b500-599b-4ffc-a35b-2cd415642d33_isr_1039673_38.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash ec81ffbad0c61be27833afa956380edd parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7268f3d7-e8c8-4892-b986-2df380e248a0_isr_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7268f3d7-e8c8-4892-b986-2df380e248a0_isr_1039769_50.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash a94baa420ceaa961f9ece52b5dee698d parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ada42bfd-7ab4-46c2-8815-52eb567e8f16_isr_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ada42bfd-7ab4-46c2-8815-52eb567e8f16_isr_1039770_49.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash d407ea5dde1e5d9c3a05ea768b9f8ec0 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:43 proc_lsst.multi:170 [INFO] [multi] job_id 17193641 proc_lsst.multi INFO: [multi] job_id 17193641 2024-03-18 04:37:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193641 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193641 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17193641 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-18 04:37:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/19bdcc4f-2c1d-4760-9172-b8e3d3c4824c_isr_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/19bdcc4f-2c1d-4760-9172-b8e3d3c4824c_isr_1039769_19.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 5579ab8cb1510a3b82ec472302c47e13 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9b845887-c59d-4320-9d03-7af7e82b442c_isr_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9b845887-c59d-4320-9d03-7af7e82b442c_isr_1039769_22.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 207650b250abd469c6724f5411a158ff parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/15c1dce2-744b-4e38-ad58-bc6576e2fb9a_isr_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/15c1dce2-744b-4e38-ad58-bc6576e2fb9a_isr_1039769_25.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 85ba211e975d665a8486f1594a15efe7 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/73ec8257-1066-4bd1-b289-976376a216bd_isr_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/73ec8257-1066-4bd1-b289-976376a216bd_isr_1039673_60.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash c852acc7e61510e38eaa623e34e3203c parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/74fdd317-4fa2-47f3-880b-4777ab9f2db6_isr_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/74fdd317-4fa2-47f3-880b-4777ab9f2db6_isr_1039769_7.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash 827ba72fa64cf5f6de5f3f2d33d301f2 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9b93d64a-eb02-49e6-a389-327b744ea61a_isr_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9b93d64a-eb02-49e6-a389-327b744ea61a_isr_1039768_38.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash f8c88f6b9c46ccb1f1035ae9e6aaa8a6 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9647c00e-fd45-4fd2-8366-ed86939456be_isr_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9647c00e-fd45-4fd2-8366-ed86939456be_isr_1039769_20.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 1ed96863a0dc57f4ea080d27138f3079 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c86d0612-1f4f-4b33-8b92-5f01c99f5bab_isr_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c86d0612-1f4f-4b33-8b92-5f01c99f5bab_isr_1039672_32.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 2b1c6f5902ae364e603beb3e5fe102c3 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4d2a6fa7-3aaf-4b6b-a39b-5d2ba1546a0b_isr_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4d2a6fa7-3aaf-4b6b-a39b-5d2ba1546a0b_isr_1039673_19.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 56e44b7876189fca73e76b64afb8a39b parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/dd063d0c-ca7e-4920-bee7-c88c5492843e_isr_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/dd063d0c-ca7e-4920-bee7-c88c5492843e_isr_1039672_7.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash 2bc47481a7de38121a65b00fd752c7cd parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f4b8c501-5061-4df3-a0b5-3b1238e24da1_isr_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f4b8c501-5061-4df3-a0b5-3b1238e24da1_isr_1039674_14.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 78f8a8d866da14b45723b21e36837fe3 parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/06ecab94-1f54-4b0e-a31c-fe6648551d06_isr_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/06ecab94-1f54-4b0e-a31c-fe6648551d06_isr_1039768_27.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 737236f3de7916b63e3750a46062d508 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/601d4732-4bb1-4ea5-981e-eac5b404c853_isr_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/601d4732-4bb1-4ea5-981e-eac5b404c853_isr_1039672_38.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash bcade7b8c313cca389691f0ef538d808 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea3a277e-fdd1-412d-b96d-7dfa54f88bf4_isr_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea3a277e-fdd1-412d-b96d-7dfa54f88bf4_isr_1039769_11.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 4f1bdff674ef0c1e561936f69ba83a29 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e980629e-5268-4105-ace6-646b80d5a53e_isr_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e980629e-5268-4105-ace6-646b80d5a53e_isr_1039770_53.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 4d85ec86e9453135e42df42ec9329daf parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8df9be2e-536d-4217-838f-ae5e873af4c7_isr_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8df9be2e-536d-4217-838f-ae5e873af4c7_isr_1039768_10.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash 23d96db9f32baa72e9411f2f6ce047c1 parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a4335ddc-a76a-421c-8c14-14556231ac9a_isr_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a4335ddc-a76a-421c-8c14-14556231ac9a_isr_1039770_23.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash d05906c6a118f061bee3ad7b391340fb parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/978dd359-a212-4098-a8bc-12ceba8320bf_isr_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/978dd359-a212-4098-a8bc-12ceba8320bf_isr_1039768_54.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 146e8d2a15bdcc5ce24e21344cde39e0 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0c6542bb-be87-438e-b4bd-645b5f2da134_isr_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0c6542bb-be87-438e-b4bd-645b5f2da134_isr_1039769_37.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 1601450f91ea57261544a1377fec6115 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d95f27c3-7a5c-4efb-b245-f6947358d59f_isr_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d95f27c3-7a5c-4efb-b245-f6947358d59f_isr_1039672_37.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash 53364e04e8b1c04e38b25db831b4f38c parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b61fbab0-8bd1-4e15-9ba4-708efd59fcdf_isr_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b61fbab0-8bd1-4e15-9ba4-708efd59fcdf_isr_1039770_44.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash 17618b1c8c75901b32520ae81d3678f0 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/3be42370-0249-454e-aef1-d45c548ccb38_isr_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/3be42370-0249-454e-aef1-d45c548ccb38_isr_1039768_52.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash 15d3c55ea63a236bb030b77e78e99685 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f4d063b4-1500-4602-ab84-b0f2cb626d6e_isr_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f4d063b4-1500-4602-ab84-b0f2cb626d6e_isr_1039672_45.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 9f76526ecbe63f189e13f188930a7f4f parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f0c8e658-f3f2-4e2e-92f6-e08af0665410_isr_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f0c8e658-f3f2-4e2e-92f6-e08af0665410_isr_1039768_35.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash ad847e9ab7e5a6ecb0817852218808a5 parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/688f7553-639e-4017-879a-eb7f47740ceb_isr_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/688f7553-639e-4017-879a-eb7f47740ceb_isr_1039770_62.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash efb6969ffa1696c0c2680e35e91dd404 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/9aebd972-58b1-4cf4-a56a-733c4155b602_isr_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/9aebd972-58b1-4cf4-a56a-733c4155b602_isr_1039672_60.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 450c9b1eedc94456e8511591301037b3 parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ffce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c516a8cc-69b1-4bfc-b4d7-c6fb2e42bdc0_isr_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c516a8cc-69b1-4bfc-b4d7-c6fb2e42bdc0_isr_1039768_11.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash b611e08efe538dc3cf59e0831a1612cc parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/bf073c24-88be-4de3-ae73-53c83cb4068a_isr_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/bf073c24-88be-4de3-ae73-53c83cb4068a_isr_1039674_15.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash cf034af50e803d421417b265ff66b5bb parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/39992be6-20d5-4b30-8d1f-ec24f6810c4e_isr_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/39992be6-20d5-4b30-8d1f-ec24f6810c4e_isr_1039672_24.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 53c9ba87714061da8487c2f1182c2468 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/938f8dd6-1c88-43dd-97be-6e052a4b06d3_isr_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/938f8dd6-1c88-43dd-97be-6e052a4b06d3_isr_1039672_47.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash d8c9e63374740e79b9793abd3b0399c0 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bef8a6bc-b24e-421b-a03c-071e29c63cdf_isr_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bef8a6bc-b24e-421b-a03c-071e29c63cdf_isr_1039672_14.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash dadc3d5b798b9af95d2944912a081487 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/cf4602d7-3634-4968-a662-aa7cd6b6b273_isr_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/cf4602d7-3634-4968-a662-aa7cd6b6b273_isr_1039770_5.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 7a0c3d17a312c1c0200b34642042531f parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6123dc43-fc8f-419a-b486-14f1f2089100_isr_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6123dc43-fc8f-419a-b486-14f1f2089100_isr_1039768_45.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash d169db04a18742b75de34ea47199b9e0 parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/30479a99-a593-462c-a06a-6dc2091f847d_isr_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/30479a99-a593-462c-a06a-6dc2091f847d_isr_1039674_2.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash 96b7e937a9078f24ff72f8ebe491904f parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f6ab3a65-6210-4b48-900e-5ee08eec9380_isr_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f6ab3a65-6210-4b48-900e-5ee08eec9380_isr_1039672_6.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 2e3766b826501ea13b685484bc2f2d26 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04cb2654-cddb-472a-b336-c7668054c4ac_isr_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04cb2654-cddb-472a-b336-c7668054c4ac_isr_1039768_22.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 134c6ffc6e5da7aabdda54ebd840530d parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:43 proc_lsst.multi:170 [INFO] [multi] job_id 17193642 proc_lsst.multi INFO: [multi] job_id 17193642 2024-03-18 04:37:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193642 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193642 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17193642 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-18 04:37:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/724ff8e4-4fa1-4a04-af3a-95a64e10ddc8_isr_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/724ff8e4-4fa1-4a04-af3a-95a64e10ddc8_isr_1039672_4.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 886bcb5a6a8f92fea16094ccf1952d97 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/036e99ae-3f21-40ab-9b8e-0ba06a6f68b5_isr_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/036e99ae-3f21-40ab-9b8e-0ba06a6f68b5_isr_1039672_3.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 7bc7d4d79ae255cf1329a88135a42cef parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/384faf82-edce-4670-bcf0-b251f94dfa80_isr_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/384faf82-edce-4670-bcf0-b251f94dfa80_isr_1039770_7.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash d6d43c9a65475c4638822bb27c44faac parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f2d90025-b6b1-43ad-a74d-fbcad14d486a_isr_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f2d90025-b6b1-43ad-a74d-fbcad14d486a_isr_1039770_19.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 093fb7f7a0a3e69b010b1330141bb932 parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c25b312f-ad2a-4657-b255-a52ac02bf333_isr_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c25b312f-ad2a-4657-b255-a52ac02bf333_isr_1039673_41.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 771bdf85ea6ccd7f0198c8a3598c0692 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/05587c87-52c6-4a31-bbc5-05eb3e0d5496_isr_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/05587c87-52c6-4a31-bbc5-05eb3e0d5496_isr_1039769_40.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash ee607042e2e72ca0578f256360181c13 parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/27665831-35b2-4471-acd7-1d805a11e28f_isr_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/27665831-35b2-4471-acd7-1d805a11e28f_isr_1039673_56.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash c5cbab7542768eae2ac45d6d14ac4faa parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/37843739-27f9-4f00-9657-38b6e8da2614_isr_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/37843739-27f9-4f00-9657-38b6e8da2614_isr_1039769_26.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 07a748e980b727917ccc2d2ccf00659d parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e1b41f9e-010c-45fa-859c-6f2b2fc1162a_isr_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e1b41f9e-010c-45fa-859c-6f2b2fc1162a_isr_1039770_26.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash 1e230f4942cde419d5c1c18b877ca7e4 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1af7fcc2-a81e-41a1-8ee6-53af5f2d3923_isr_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1af7fcc2-a81e-41a1-8ee6-53af5f2d3923_isr_1039768_12.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash b72cff528ab8f47382bb15f80348af48 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/feeb7895-a082-4666-930e-34726bc052c7_isr_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/feeb7895-a082-4666-930e-34726bc052c7_isr_1039672_29.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash e0197648506fb0213d140beec3243a8b parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1e70e906-6058-440c-8801-1f7a257b55d0_isr_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1e70e906-6058-440c-8801-1f7a257b55d0_isr_1039673_62.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 627a22e664c178333f7fd79072c884c4 parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/236b6243-44b2-4ecf-8bea-59db07527fbf_isr_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/236b6243-44b2-4ecf-8bea-59db07527fbf_isr_1039768_41.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 5655e5189a1afc94b51801998bd149d6 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/3644b0d3-f098-4047-97a1-716b67087bfd_isr_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/3644b0d3-f098-4047-97a1-716b67087bfd_isr_1039672_57.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 4e6d65fd23b1680158800aa91cf7811c parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/26960b24-44df-42a9-8c2e-a743b70568f9_isr_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/26960b24-44df-42a9-8c2e-a743b70568f9_isr_1039770_43.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 7d38263dcaa394a87ccd02d3a87ae6c7 parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ef31bc9d-a4a4-4bdb-b30f-a6c417d4ff39_isr_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ef31bc9d-a4a4-4bdb-b30f-a6c417d4ff39_isr_1039672_25.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash cdf75693cad8965b3591f16f378f8896 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/8b5e51c2-5a06-4a9a-8860-3bcd739a395c_isr_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/8b5e51c2-5a06-4a9a-8860-3bcd739a395c_isr_1039770_13.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 7cf948f84808a05ac7a3fdbdda775c82 parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cc3cabc1-a8a7-47c8-ba9a-a5abbcb617e3_isr_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cc3cabc1-a8a7-47c8-ba9a-a5abbcb617e3_isr_1039672_33.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 16ea490e7d544558c8138fa220c99548 parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/34c7bbf4-23f0-46e6-93cb-53db66425f1e_isr_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/34c7bbf4-23f0-46e6-93cb-53db66425f1e_isr_1039768_39.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 5716012de01ac771e0a64ec45f62be38 parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c506cf56-2725-468c-ba9b-6c650a2109d7_isr_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c506cf56-2725-468c-ba9b-6c650a2109d7_isr_1039674_18.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash c7d8f108aa9178eee03a0752fd305ccd parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c9afc6de-8162-4be0-a727-ce3a003e273b_isr_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c9afc6de-8162-4be0-a727-ce3a003e273b_isr_1039769_4.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 00716c9a7dd060f0c8347dc89efd1424 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0434aa7a-de16-4aa1-bc93-227cbc70516d_isr_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0434aa7a-de16-4aa1-bc93-227cbc70516d_isr_1039770_39.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash 2d6290122990ace408b008297d6b7acc parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/cb3516fc-3924-4db3-b715-60b84c20f8b1_isr_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/cb3516fc-3924-4db3-b715-60b84c20f8b1_isr_1039768_19.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash d68078498264dad068b0e420f51d16b2 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0d9f564f-0c5c-4e42-9087-779547974c6f_isr_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0d9f564f-0c5c-4e42-9087-779547974c6f_isr_1039769_12.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash ae7b7b171d08cbd592f0de95094ad763 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e856f87b-dd66-451f-aa09-07d2bb5c0d58_isr_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e856f87b-dd66-451f-aa09-07d2bb5c0d58_isr_1039768_21.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash b9688632636a02c379135aad5fd25919 parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cbe4a2bf-3c64-41c2-90cb-228b7e8a704a_isr_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cbe4a2bf-3c64-41c2-90cb-228b7e8a704a_isr_1039674_39.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 515d4211514b2056e9ff758d43af24e5 parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bcea016d-47f6-4546-a0b9-9c947c8c1432_isr_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bcea016d-47f6-4546-a0b9-9c947c8c1432_isr_1039769_3.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash ba305b3d41fd87aef9cea0e9bf7cb14d parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aad4d5c5-5ea1-49eb-8608-85c00b2ae714_isr_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aad4d5c5-5ea1-49eb-8608-85c00b2ae714_isr_1039674_41.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 061274326a9ed7693643872e02bff783 parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/056e18dd-25b1-4d4a-a354-c46b3f548450_isr_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/056e18dd-25b1-4d4a-a354-c46b3f548450_isr_1039673_23.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash ce006f857edf040775aeaef5e049bd38 parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8a6def3e-7943-478f-af3f-4d678eef3a13_isr_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8a6def3e-7943-478f-af3f-4d678eef3a13_isr_1039672_11.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash eb3a668eebcbb50de018e97fee0bd5a2 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4217ed90-0996-4ac6-bf56-a188159cc873_isr_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4217ed90-0996-4ac6-bf56-a188159cc873_isr_1039674_3.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash 54d2e79f3be85527cea1d0fce74eeedd parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f3ee55bc-d2b4-40ec-b915-07ee069e2719_isr_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f3ee55bc-d2b4-40ec-b915-07ee069e2719_isr_1039674_11.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 0c2eeb3c843816d9dadfd062be43f7b1 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/2503fd28-0233-4b8e-b6ec-6e9828efba3e_isr_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/2503fd28-0233-4b8e-b6ec-6e9828efba3e_isr_1039769_51.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 47b8056bfa3e79a288381b825b81f9cb parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8320ecc7-1d39-45a4-a06f-f6430653717c_isr_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8320ecc7-1d39-45a4-a06f-f6430653717c_isr_1039672_5.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash 20fd64f63aaf46ba99586d38f32347db parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1c5d3d3d-52a7-4077-8b84-8f929290e302_isr_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1c5d3d3d-52a7-4077-8b84-8f929290e302_isr_1039770_14.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash 536cb5292176c1db8ac6a3e205f885f3 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a2825217-ad35-4e11-a2ec-893846cfc618_isr_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a2825217-ad35-4e11-a2ec-893846cfc618_isr_1039672_53.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 7953789804f2bbf536a3492562c051fa parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/54edcf56-2ac6-46b2-bc1c-3f970c2b0129_isr_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/54edcf56-2ac6-46b2-bc1c-3f970c2b0129_isr_1039672_31.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash aad2e7f869adea2f8bafbb07bb2025fc parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a27885a2-e3cd-41bf-8233-7d373676e0b2_isr_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a27885a2-e3cd-41bf-8233-7d373676e0b2_isr_1039673_28.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash 7631f3c52bd227c4e59837bc619ca758 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/19897253-7d6a-4b4c-8573-4ea792f17c73_isr_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/19897253-7d6a-4b4c-8573-4ea792f17c73_isr_1039673_55.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash dcf9528c3ae0dc7be11c6e62c913bbe8 parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/57706252-3ead-439a-9a25-6cebc4e101f7_isr_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/57706252-3ead-439a-9a25-6cebc4e101f7_isr_1039672_28.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash 1a40c8719a275b090816d5745f51044c parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:44 proc_lsst.multi:170 [INFO] [multi] job_id 17193643 proc_lsst.multi INFO: [multi] job_id 17193643 2024-03-18 04:37:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193643 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193643 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17193643 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-18 04:37:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-18 04:37:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/565cb7d4-3066-4006-8b89-1399cc8ea6a2_isr_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/565cb7d4-3066-4006-8b89-1399cc8ea6a2_isr_1039769_38.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash 499e1c78784881617c3ecfa036174c53 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0a5d8a0e-4ba8-4fa9-94b0-6ae11d99119c_isr_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0a5d8a0e-4ba8-4fa9-94b0-6ae11d99119c_isr_1039769_2.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 00b62409ba4f0ebd936a42612348c787 parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/a51e1f1c-1683-4289-86d5-b5535617aa86_isr_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/a51e1f1c-1683-4289-86d5-b5535617aa86_isr_1039768_9.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash 49f63197f09c835643cc6bf3db562f47 parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7405b285-f53a-4386-ab3a-7090b23482a9_isr_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7405b285-f53a-4386-ab3a-7090b23482a9_isr_1039673_47.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 3a187aa4d5cfad55638035b383a7dadf parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4f3beb2a-50c4-4384-88f4-b7dd5a0732c7_isr_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4f3beb2a-50c4-4384-88f4-b7dd5a0732c7_isr_1039674_45.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 9983cea0c12b9fe1f0135e631feb1592 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fe9741f8-7784-499b-81ca-22624fea81e5_isr_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fe9741f8-7784-499b-81ca-22624fea81e5_isr_1039770_8.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash a1751a00a9d659edd43effc37cf879b9 parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f439c2d8-927f-4a60-b403-350ce4a82bce_isr_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f439c2d8-927f-4a60-b403-350ce4a82bce_isr_1039673_1.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 622d745d45fbd3ce576bf9003b847243 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/aa0545e6-e57b-488d-8b9e-6a288b1e8a7c_isr_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/aa0545e6-e57b-488d-8b9e-6a288b1e8a7c_isr_1039672_30.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash c20dbcecc1f99104ba64236a239694d1 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cf152fab-3e7e-4915-930d-dffe3fee3635_isr_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cf152fab-3e7e-4915-930d-dffe3fee3635_isr_1039674_29.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash e7980ac62042d8904d293f08614ddb91 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5728a37-6cce-4c8e-a1a6-4f4cda981a66_isr_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5728a37-6cce-4c8e-a1a6-4f4cda981a66_isr_1039769_23.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash 4708b32c7aa54f7b051e50b276413076 parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d836b79c-d920-4683-b1eb-96200f2412c5_isr_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d836b79c-d920-4683-b1eb-96200f2412c5_isr_1039674_44.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 0746f0fca9a658385577ba683b24886d parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bbc446fd-7c38-4e38-a95a-15526254bd26_isr_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bbc446fd-7c38-4e38-a95a-15526254bd26_isr_1039672_41.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash d99629aa790f115d3fdb62c95069c337 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/d871645a-480b-49a9-a9ed-3ead40782998_isr_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/d871645a-480b-49a9-a9ed-3ead40782998_isr_1039770_51.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash 2656fd2de2b856ed8246056965164d8f parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2140e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/841195da-8925-4dca-b863-dcc9f700c8af_isr_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/841195da-8925-4dca-b863-dcc9f700c8af_isr_1039674_7.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 8ec037821d46e12ce624cdc17a7078a9 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c9ab1468-45b5-4cff-b40b-f47f62dc5efc_isr_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c9ab1468-45b5-4cff-b40b-f47f62dc5efc_isr_1039672_49.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash a1906a808b5abe1eb91d169de37b040d parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b9f4a176-ffe9-479f-877a-dbd6cc979208_isr_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b9f4a176-ffe9-479f-877a-dbd6cc979208_isr_1039770_40.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash e0d32ddff10fc32158dfed739ed786ab parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d7cd9e78-719b-47ab-80d6-971d2a6db9a6_isr_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d7cd9e78-719b-47ab-80d6-971d2a6db9a6_isr_1039768_1.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash 6506b1b11aa7d5b12e969a2ed5a138ee parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/69f91105-4c3d-4fc5-aab4-8dc12d66d0f8_isr_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/69f91105-4c3d-4fc5-aab4-8dc12d66d0f8_isr_1039673_13.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 822a9801eefe2dea61f5a6f581f8f1f3 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2145e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dabb7daf-ccdc-45bc-829d-39dbe0bf9128_isr_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dabb7daf-ccdc-45bc-829d-39dbe0bf9128_isr_1039769_31.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash d2efd2cbf426412ceaf094bd20e67f60 parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2142c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/6cb39efc-72a0-4b9c-90f2-68d05aadba5e_isr_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/6cb39efc-72a0-4b9c-90f2-68d05aadba5e_isr_1039674_56.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 64ef55a962b2662572b0f862656e66ac parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2158a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c2e8544b-81df-416a-92f9-903e118ab3de_isr_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c2e8544b-81df-416a-92f9-903e118ab3de_isr_1039770_45.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash b2a80151df9315fa71c746cc0aca316c parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/405621de-3c47-4624-b5cf-74fbafa6de30_isr_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/405621de-3c47-4624-b5cf-74fbafa6de30_isr_1039769_17.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash 5295a6d5dc5b3fc448c890aeab68c838 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-18 04:37:44 proc_lsst.multi:170 [INFO] [multi] job_id 17193644 proc_lsst.multi INFO: [multi] job_id 17193644 2024-03-18 04:37:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193644 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193644 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17193644 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-18 04:37:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-18 04:37:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7a375cfb-3ec6-4e16-95a4-a0e79305c6a1_isr_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7a375cfb-3ec6-4e16-95a4-a0e79305c6a1_isr_1039672_61.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash c557038d2a50d119281a7d269e3e6e3e parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/46927fca-bb27-4eba-af2d-c681b455818f_isr_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/46927fca-bb27-4eba-af2d-c681b455818f_isr_1039770_17.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash e30246062ae8f79aa996a5a40e1b30d4 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/521b2b1a-2ce3-4b8b-8f5b-4c5e9519fb08_isr_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/521b2b1a-2ce3-4b8b-8f5b-4c5e9519fb08_isr_1039674_59.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash 7f76bc86b34a9b0d9f5c5f5267438237 parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2165c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/2c407beb-96cb-47eb-9f50-27f15e813017_isr_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/2c407beb-96cb-47eb-9f50-27f15e813017_isr_1039674_38.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash d09c431c1b53f9035a2d0503bc797ae4 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f8c22207-4fad-442a-888e-42905ffa35f3_isr_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f8c22207-4fad-442a-888e-42905ffa35f3_isr_1039673_21.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash f71e462e30134014546d6202e50364eb parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54a2c220-4834-4ed1-9aeb-d57a247c8908_isr_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54a2c220-4834-4ed1-9aeb-d57a247c8908_isr_1039770_1.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash efb99b472b3d9f4a2ea9ffa4f32a5f57 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/952614bc-0142-4017-aecb-e8222e0a7eca_isr_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/952614bc-0142-4017-aecb-e8222e0a7eca_isr_1039674_42.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 6207575b3031cffc214bad9e32fb0bb1 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/eeb43dbd-8191-44e0-87fe-75daca31486e_isr_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/eeb43dbd-8191-44e0-87fe-75daca31486e_isr_1039770_61.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash a74cc29f10f89da1b0520739b550d92a parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2171a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9ca6fcce-72d1-4f55-aef0-a3165ee14cdd_isr_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9ca6fcce-72d1-4f55-aef0-a3165ee14cdd_isr_1039673_51.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 023d0b8f0224792ed5d3f8991a2c586d parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2174c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3922ca5f-525c-4295-8be9-8cfa91224afb_isr_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3922ca5f-525c-4295-8be9-8cfa91224afb_isr_1039770_22.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 87b1ffe450602789016fa29356657359 parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/40e9e1ff-8fb4-40eb-98b7-989728bc204b_isr_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/40e9e1ff-8fb4-40eb-98b7-989728bc204b_isr_1039673_39.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash 69287b1c2d65087b25db94dee4ebdb8e parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d8e906a-a863-4399-8a25-789d20afa2e2_isr_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d8e906a-a863-4399-8a25-789d20afa2e2_isr_1039673_50.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash a96b202703327dbee2db7ccf460fee2a parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8d6d226f-3e70-4631-a521-cf4d0574460e_isr_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8d6d226f-3e70-4631-a521-cf4d0574460e_isr_1039674_31.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash aeaa5df4365c502b38d9b7ca4e88443c parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1ba94237-d74f-4086-8584-cc23eeae4b89_isr_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1ba94237-d74f-4086-8584-cc23eeae4b89_isr_1039768_25.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash e192be1e180bc0806f2efd0719064d6d parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0740e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0e90c774-8304-4d33-b735-607eb84cbe24_isr_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0e90c774-8304-4d33-b735-607eb84cbe24_isr_1039770_18.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash cc4f5c7e6845f02048f7bceb6acee210 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0745e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/72a0ad8b-c54f-4363-80c8-217102c29c9f_isr_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/72a0ad8b-c54f-4363-80c8-217102c29c9f_isr_1039768_37.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 803cd429672e2391732863a1cd4a2e51 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4f487394-45c1-4679-a8c6-cc30adea4c00_isr_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4f487394-45c1-4679-a8c6-cc30adea4c00_isr_1039673_40.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash 30945af285df5081cf2d6aaeaa33370e parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8047c5b7-e005-416f-9993-0ec658ab944f_isr_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8047c5b7-e005-416f-9993-0ec658ab944f_isr_1039672_51.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 9eeaf9ef92bd1c3431cceca09d992ca3 parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/689a65a4-e35d-4bbf-9b7b-62e27d4291d5_isr_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/689a65a4-e35d-4bbf-9b7b-62e27d4291d5_isr_1039673_3.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash 7c5eb87fcb9b72df33ccdbef5fee699d parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/347d60bf-472a-48db-86e3-1f28fe943904_isr_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/347d60bf-472a-48db-86e3-1f28fe943904_isr_1039770_31.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash 0f9f8a2408210df19f4f5bc892d87fdb parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1500fdcd-486c-4cb5-8919-529f58a18a47_isr_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1500fdcd-486c-4cb5-8919-529f58a18a47_isr_1039674_32.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-18 04:37:45 proc_lsst.multi:170 [INFO] [multi] job_id 17193645 proc_lsst.multi INFO: [multi] job_id 17193645 2024-03-18 04:37:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193645 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193645 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17193645 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-18 04:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:45 proc_lsst.multi:170 [INFO] [multi] job_id 17193646 proc_lsst.multi INFO: [multi] job_id 17193646 2024-03-18 04:37:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193646 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193646 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17193646 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-18 04:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:45 proc_lsst.multi:170 [INFO] [multi] job_id 17193647 proc_lsst.multi INFO: [multi] job_id 17193647 2024-03-18 04:37:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193647 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193647 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17193647 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-18 04:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:45 proc_lsst.multi:170 [INFO] [multi] job_id 17193648 proc_lsst.multi INFO: [multi] job_id 17193648 2024-03-18 04:37:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193648 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193648 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17193648 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-18 04:37:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:46 proc_lsst.multi:170 [INFO] [multi] job_id 17193649 proc_lsst.multi INFO: [multi] job_id 17193649 2024-03-18 04:37:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193649 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193649 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17193649 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-18 04:37:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:46 proc_lsst.multi:170 [INFO] [multi] job_id 17193650 proc_lsst.multi INFO: [multi] job_id 17193650 2024-03-18 04:37:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193650 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193650 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17193650 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-18 04:37:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:46 proc_lsst.multi:170 [INFO] [multi] job_id 17193651 proc_lsst.multi INFO: [multi] job_id 17193651 2024-03-18 04:37:46 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:46 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193651 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193651 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17193651 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-18 04:37:46 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:46 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:46 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:46 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:46 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 17193652 proc_lsst.multi INFO: [multi] job_id 17193652 2024-03-18 04:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193652 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193652 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17193652 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-18 04:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 17193653 proc_lsst.multi INFO: [multi] job_id 17193653 2024-03-18 04:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193653 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193653 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17193653 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-18 04:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:47 proc_lsst.multi:170 [INFO] [multi] job_id 17193654 proc_lsst.multi INFO: [multi] job_id 17193654 2024-03-18 04:37:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193654 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193654 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17193654 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-18 04:37:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193655 proc_lsst.multi INFO: [multi] job_id 17193655 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193655 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193655 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17193655 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193656 proc_lsst.multi INFO: [multi] job_id 17193656 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193656 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193656 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17193656 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193657 proc_lsst.multi INFO: [multi] job_id 17193657 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193657 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193657 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17193657 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193658 proc_lsst.multi INFO: [multi] job_id 17193658 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193658 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193658 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17193658 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193659 proc_lsst.multi INFO: [multi] job_id 17193659 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193659 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193659 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17193659 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193660 proc_lsst.multi INFO: [multi] job_id 17193660 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193660 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193660 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17193660 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193661 proc_lsst.multi INFO: [multi] job_id 17193661 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193661 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193661 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17193661 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193662 proc_lsst.multi INFO: [multi] job_id 17193662 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193662 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193662 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17193662 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:48 proc_lsst.multi:170 [INFO] [multi] job_id 17193663 proc_lsst.multi INFO: [multi] job_id 17193663 2024-03-18 04:37:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 04:37:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193663 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193663 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17193663 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-18 04:37:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 04:37:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193664 proc_lsst.multi INFO: [multi] job_id 17193664 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193664 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193664 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17193664 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193665 proc_lsst.multi INFO: [multi] job_id 17193665 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193665 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193665 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17193665 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193666 proc_lsst.multi INFO: [multi] job_id 17193666 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17193666 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17193666 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17193666 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193667 proc_lsst.multi INFO: [multi] job_id 17193667 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17193667 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17193667 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17193667 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193668 proc_lsst.multi INFO: [multi] job_id 17193668 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17193668 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17193668 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17193668 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/32 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 60 more blocks parsl.executors.status_handling INFO: Scaling out by 60 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:49 proc_lsst.multi:170 [INFO] [multi] job_id 17193669 proc_lsst.multi INFO: [multi] job_id 17193669 2024-03-18 04:37:49 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:49 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17193669 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17193669 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17193669 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-18 04:37:49 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:49 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:49 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:49 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:49 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193670 proc_lsst.multi INFO: [multi] job_id 17193670 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17193670 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17193670 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17193670 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193671 proc_lsst.multi INFO: [multi] job_id 17193671 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17193671 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17193671 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17193671 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193672 proc_lsst.multi INFO: [multi] job_id 17193672 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193672 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193672 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17193672 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193673 proc_lsst.multi INFO: [multi] job_id 17193673 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193673 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193673 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17193673 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193674 proc_lsst.multi INFO: [multi] job_id 17193674 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193674 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193674 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17193674 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193675 proc_lsst.multi INFO: [multi] job_id 17193675 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193675 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193675 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17193675 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193676 proc_lsst.multi INFO: [multi] job_id 17193676 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193676 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193676 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17193676 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:50 proc_lsst.multi:170 [INFO] [multi] job_id 17193677 proc_lsst.multi INFO: [multi] job_id 17193677 2024-03-18 04:37:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193677 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193677 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17193677 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-18 04:37:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:50 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193678 proc_lsst.multi INFO: [multi] job_id 17193678 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193678 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193678 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17193678 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193680 proc_lsst.multi INFO: [multi] job_id 17193680 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193680 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193680 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17193680 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193681 proc_lsst.multi INFO: [multi] job_id 17193681 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193681 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193681 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17193681 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193682 proc_lsst.multi INFO: [multi] job_id 17193682 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193682 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193682 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17193682 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193683 proc_lsst.multi INFO: [multi] job_id 17193683 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193683 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193683 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17193683 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:51 proc_lsst.multi:170 [INFO] [multi] job_id 17193684 proc_lsst.multi INFO: [multi] job_id 17193684 2024-03-18 04:37:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193684 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193684 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17193684 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-18 04:37:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:51 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:52 proc_lsst.multi:170 [INFO] [multi] job_id 17193685 proc_lsst.multi INFO: [multi] job_id 17193685 2024-03-18 04:37:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193685 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193685 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17193685 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-18 04:37:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:52 proc_lsst.multi:170 [INFO] [multi] job_id 17193686 proc_lsst.multi INFO: [multi] job_id 17193686 2024-03-18 04:37:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193686 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193686 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17193686 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-18 04:37:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:52 proc_lsst.multi:170 [INFO] [multi] job_id 17193687 proc_lsst.multi INFO: [multi] job_id 17193687 2024-03-18 04:37:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193687 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193687 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17193687 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-18 04:37:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:52 proc_lsst.multi:170 [INFO] [multi] job_id 17193688 proc_lsst.multi INFO: [multi] job_id 17193688 2024-03-18 04:37:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193688 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193688 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17193688 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-18 04:37:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:52 proc_lsst.multi:170 [INFO] [multi] job_id 17193689 proc_lsst.multi INFO: [multi] job_id 17193689 2024-03-18 04:37:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193689 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193689 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17193689 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-18 04:37:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:52 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:53 proc_lsst.multi:170 [INFO] [multi] job_id 17193690 proc_lsst.multi INFO: [multi] job_id 17193690 2024-03-18 04:37:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193690 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193690 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17193690 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-18 04:37:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:53 proc_lsst.multi:170 [INFO] [multi] job_id 17193691 proc_lsst.multi INFO: [multi] job_id 17193691 2024-03-18 04:37:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193691 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193691 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17193691 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-18 04:37:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:53 proc_lsst.multi:170 [INFO] [multi] job_id 17193692 proc_lsst.multi INFO: [multi] job_id 17193692 2024-03-18 04:37:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17193692 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17193692 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17193692 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-18 04:37:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:53 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-18 04:37:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 17193693 proc_lsst.multi INFO: [multi] job_id 17193693 2024-03-18 04:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193693 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193693 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17193693 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-18 04:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 17193694 proc_lsst.multi INFO: [multi] job_id 17193694 2024-03-18 04:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193694 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193694 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17193694 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-18 04:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:54 proc_lsst.multi:170 [INFO] [multi] job_id 17193695 proc_lsst.multi INFO: [multi] job_id 17193695 2024-03-18 04:37:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193695 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193695 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17193695 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-18 04:37:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:54 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-18 04:37:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 17193696 proc_lsst.multi INFO: [multi] job_id 17193696 2024-03-18 04:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193696 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193696 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17193696 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-18 04:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 17193697 proc_lsst.multi INFO: [multi] job_id 17193697 2024-03-18 04:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193697 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193697 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17193697 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-18 04:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 17193698 proc_lsst.multi INFO: [multi] job_id 17193698 2024-03-18 04:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193698 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193698 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17193698 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-18 04:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:55 proc_lsst.multi:170 [INFO] [multi] job_id 17193699 proc_lsst.multi INFO: [multi] job_id 17193699 2024-03-18 04:37:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193699 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193699 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17193699 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-18 04:37:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:55 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-18 04:37:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193700 proc_lsst.multi INFO: [multi] job_id 17193700 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193700 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193700 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17193700 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193701 proc_lsst.multi INFO: [multi] job_id 17193701 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193701 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193701 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17193701 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193702 proc_lsst.multi INFO: [multi] job_id 17193702 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193702 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193702 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17193702 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193703 proc_lsst.multi INFO: [multi] job_id 17193703 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193703 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193703 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17193703 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193704 proc_lsst.multi INFO: [multi] job_id 17193704 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193704 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193704 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17193704 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:56 proc_lsst.multi:170 [INFO] [multi] job_id 17193705 proc_lsst.multi INFO: [multi] job_id 17193705 2024-03-18 04:37:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193705 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193705 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17193705 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-18 04:37:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:56 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-18 04:37:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193706 proc_lsst.multi INFO: [multi] job_id 17193706 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193706 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193706 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17193706 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8318f1dec18f7ad154bce9a092abe0a1 with result from task 3 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193707 proc_lsst.multi INFO: [multi] job_id 17193707 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193707 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193707 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17193707 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash abdc1b5f540abf0c46fc334f817deaf4 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d9c40a07-c3d8-438b-b058-eec38eb68f1b_characterizeImage_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d9c40a07-c3d8-438b-b058-eec38eb68f1b_characterizeImage_1039768_31.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/daf104bf-ba84-4a6c-a360-de03738ff476_isr_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/daf104bf-ba84-4a6c-a360-de03738ff476_isr_1039768_31.stderr 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193708 proc_lsst.multi INFO: [multi] job_id 17193708 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193708 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193708 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17193708 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193709 proc_lsst.multi INFO: [multi] job_id 17193709 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193709 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193709 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17193709 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193710 proc_lsst.multi INFO: [multi] job_id 17193710 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193710 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193710 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17193710 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:57 proc_lsst.multi:170 [INFO] [multi] job_id 17193711 proc_lsst.multi INFO: [multi] job_id 17193711 2024-03-18 04:37:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193711 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193711 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17193711 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-18 04:37:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:57 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-18 04:37:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:58 proc_lsst.multi:170 [INFO] [multi] job_id 17193712 proc_lsst.multi INFO: [multi] job_id 17193712 2024-03-18 04:37:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193712 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193712 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17193712 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-18 04:37:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:58 proc_lsst.multi:170 [INFO] [multi] job_id 17193713 proc_lsst.multi INFO: [multi] job_id 17193713 2024-03-18 04:37:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 04:37:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:58 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193713 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193713 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17193713 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-18 04:37:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:58 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-18 04:37:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bbda8f47d16be3708fb57da9aff5551 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 37ef0ec04d459dc3fbe71834452f562f parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1092e414-cfa2-45cb-9ee0-b3eb99d12407_characterizeImage_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1092e414-cfa2-45cb-9ee0-b3eb99d12407_characterizeImage_1039672_36.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a3b444a9-6a91-4c75-a5ad-69ed1fcdf431_isr_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a3b444a9-6a91-4c75-a5ad-69ed1fcdf431_isr_1039672_36.stderr 2024-03-18 04:37:59 proc_lsst.multi:170 [INFO] [multi] job_id 17193714 proc_lsst.multi INFO: [multi] job_id 17193714 2024-03-18 04:37:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 04:37:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193714 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193714 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17193714 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-18 04:37:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:37:59 proc_lsst.multi:170 [INFO] [multi] job_id 17193715 proc_lsst.multi INFO: [multi] job_id 17193715 2024-03-18 04:37:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 04:37:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:59 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193715 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193715 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17193715 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-18 04:37:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:37:59 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:37:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-18 04:37:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:37:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:00 proc_lsst.multi:170 [INFO] [multi] job_id 17193716 proc_lsst.multi INFO: [multi] job_id 17193716 2024-03-18 04:38:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 04:38:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193716 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193716 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17193716 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-18 04:38:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:00 proc_lsst.multi:170 [INFO] [multi] job_id 17193717 proc_lsst.multi INFO: [multi] job_id 17193717 2024-03-18 04:38:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 04:38:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:00 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193717 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193717 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17193717 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-18 04:38:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:00 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-18 04:38:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:01 proc_lsst.multi:170 [INFO] [multi] job_id 17193718 proc_lsst.multi INFO: [multi] job_id 17193718 2024-03-18 04:38:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 04:38:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193718 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193718 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17193718 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-18 04:38:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:01 proc_lsst.multi:170 [INFO] [multi] job_id 17193719 proc_lsst.multi INFO: [multi] job_id 17193719 2024-03-18 04:38:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 04:38:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:01 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193719 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193719 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17193719 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-18 04:38:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:01 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-18 04:38:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:02 proc_lsst.multi:170 [INFO] [multi] job_id 17193720 proc_lsst.multi INFO: [multi] job_id 17193720 2024-03-18 04:38:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 04:38:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193720 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193720 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17193720 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-18 04:38:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:02 proc_lsst.multi:170 [INFO] [multi] job_id 17193721 proc_lsst.multi INFO: [multi] job_id 17193721 2024-03-18 04:38:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 04:38:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:02 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193721 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193721 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17193721 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-18 04:38:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:02 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-18 04:38:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:03 proc_lsst.multi:170 [INFO] [multi] job_id 17193722 proc_lsst.multi INFO: [multi] job_id 17193722 2024-03-18 04:38:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193722 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193722 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17193722 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-18 04:38:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:03 proc_lsst.multi:170 [INFO] [multi] job_id 17193723 proc_lsst.multi INFO: [multi] job_id 17193723 2024-03-18 04:38:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-18 04:38:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:03 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193723 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193723 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17193723 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-18 04:38:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:03 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-18 04:38:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:04 proc_lsst.multi:170 [INFO] [multi] job_id 17193724 proc_lsst.multi INFO: [multi] job_id 17193724 2024-03-18 04:38:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-18 04:38:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193724 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193724 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17193724 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-18 04:38:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:04 proc_lsst.multi:170 [INFO] [multi] job_id 17193725 proc_lsst.multi INFO: [multi] job_id 17193725 2024-03-18 04:38:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-18 04:38:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193725 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193725 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17193725 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-18 04:38:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:04 proc_lsst.multi:170 [INFO] [multi] job_id 17193726 proc_lsst.multi INFO: [multi] job_id 17193726 2024-03-18 04:38:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-18 04:38:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193726 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193726 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17193726 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-18 04:38:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:04 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-18 04:38:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:05 proc_lsst.multi:170 [INFO] [multi] job_id 17193727 proc_lsst.multi INFO: [multi] job_id 17193727 2024-03-18 04:38:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-18 04:38:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193727 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193727 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17193727 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-18 04:38:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:05 proc_lsst.multi:170 [INFO] [multi] job_id 17193728 proc_lsst.multi INFO: [multi] job_id 17193728 2024-03-18 04:38:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-18 04:38:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193728 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193728 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17193728 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-18 04:38:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-18 04:38:05 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-18 04:38:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-18 04:38:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 10.64.129.9,198.48.92.26,169.254.95.120,127.0.0.1,10.64.65.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54507 --result_port=54441 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-18 04:38:05 proc_lsst.multi:170 [INFO] [multi] job_id 17193729 proc_lsst.multi INFO: [multi] job_id 17193729 2024-03-18 04:38:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-18 04:38:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-18 04:38:05 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17193729 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17193729 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17193729 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 48 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f47f359df5b176e8fda2ba2ed78b5713 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 4808d5e5f8ae0182bed8918210634ad8 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/554f5e58-dff4-4dea-95fe-3edd6a8e2909_isr_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/554f5e58-dff4-4dea-95fe-3edd6a8e2909_isr_1039768_2.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c48d0ff84ecce9d583d5d5dc42e8494b with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash dad5a2215d44227c1f701a7ac5c16a02 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b90f7d97-08a2-41d5-ab83-b718ae82e472_characterizeImage_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b90f7d97-08a2-41d5-ab83-b718ae82e472_characterizeImage_1039768_18.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/64b27773-9979-41d8-94cf-a9186b91a996_isr_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/64b27773-9979-41d8-94cf-a9186b91a996_isr_1039768_18.stderr 2024-03-18 04:38:15 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:38:15 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:16 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:16 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:16 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:16 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:16 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:17 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:18 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:18 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:18 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:18 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:19 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:19 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:19 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:19 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:20 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:20 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:21 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:21 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:21 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:22 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:22 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:23 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:23 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:24 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:24 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:25 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:25 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bd89dc9235be354599f42651fb268b4 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 96ca5c2ae5d2d0ee6f937acee12eb4dc parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0b4acbd4-b418-4a27-94ae-4606e7830e3e_characterizeImage_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0b4acbd4-b418-4a27-94ae-4606e7830e3e_characterizeImage_1039674_30.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d37b4d07-e081-4f1b-8312-36cdfd6d8fe8_isr_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d37b4d07-e081-4f1b-8312-36cdfd6d8fe8_isr_1039674_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:26 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d7616fa5068d69d70995a3698f8171e with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 8d4843352daa91e237dde976baf232ce parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/202cc26a-8cea-4094-b2e9-57cfbaeb25e0_characterizeImage_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/202cc26a-8cea-4094-b2e9-57cfbaeb25e0_characterizeImage_1039768_24.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8e6b83f8-b368-44da-b9a7-56693f8a5204_isr_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8e6b83f8-b368-44da-b9a7-56693f8a5204_isr_1039768_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:26 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:26 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:27 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:27 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:27 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:27 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:28 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:28 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:28 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:28 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:29 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:29 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:29 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:30 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:31 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe7f89902812612ed9b306cf50f28d4 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash f5e836b0f93fee4fad4f087b7911a2d5 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/44bb32a1-d38d-490d-82ea-36481bca8907_isr_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/44bb32a1-d38d-490d-82ea-36481bca8907_isr_1039672_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:32 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:32 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:33 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 596a8dccad723d4e3a35dae6b99af6a2 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash d69ffd47cc575427aa3e221a138cce49 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/877fa3ce-1b77-4cfe-b3c7-1e461cf118b1_characterizeImage_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/877fa3ce-1b77-4cfe-b3c7-1e461cf118b1_characterizeImage_1039672_17.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b4530209-2169-4810-9481-4f906967e8e0_isr_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b4530209-2169-4810-9481-4f906967e8e0_isr_1039672_17.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b469159776a289784966164a76c6e6b with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash cb305210ec2c1ad0faeaf282ef9d8769 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3a416cc-6fb9-4f90-9af6-824a4c1387e9_characterizeImage_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3a416cc-6fb9-4f90-9af6-824a4c1387e9_characterizeImage_1039769_43.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b289a6f3-f624-4560-be57-2fc80053fe6f_isr_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b289a6f3-f624-4560-be57-2fc80053fe6f_isr_1039769_43.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53adc3ca19c67f07c06e108394c2a3cf with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 3dee6dda9e979f242d7e82c6136b728f parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9127a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f7dac3e-54b1-4071-ab4e-3c9dc52efdf8_characterizeImage_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f7dac3e-54b1-4071-ab4e-3c9dc52efdf8_characterizeImage_1039673_14.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d5b37484-eb91-45a2-accb-cad3a0bd642a_isr_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d5b37484-eb91-45a2-accb-cad3a0bd642a_isr_1039673_14.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4041c5c3fb4eddede4c61026860245b with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 2aa6d5e83f3ebeea6e8dd188834af52f parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4fa7f347-445a-4709-b1b8-fc0659ee8cdf_characterizeImage_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4fa7f347-445a-4709-b1b8-fc0659ee8cdf_characterizeImage_1039770_48.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/9cfe5630-2c9c-4030-a09c-49d082d877cb_isr_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/9cfe5630-2c9c-4030-a09c-49d082d877cb_isr_1039770_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:34 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2016c2ba40fac7a6b3de080ca05116b with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash df2a99ccf34b2b7a39205adcedcc50bc parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f87ffc45-7aa8-4471-9546-2665217375b8_characterizeImage_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f87ffc45-7aa8-4471-9546-2665217375b8_characterizeImage_1039768_15.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/994e98c4-c041-41a3-900d-906293916b4c_isr_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/994e98c4-c041-41a3-900d-906293916b4c_isr_1039768_15.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2580c7fe1c35d2938b44b2891971dae0 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 7b9da68ecabd1c79794d0ba6dcfd9e69 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/2aab8548-f9ee-42f5-9726-3fb1f283b5fc_characterizeImage_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/2aab8548-f9ee-42f5-9726-3fb1f283b5fc_characterizeImage_1039768_58.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/255d4340-7639-47bd-84ce-e4ac52c782d0_isr_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/255d4340-7639-47bd-84ce-e4ac52c782d0_isr_1039768_58.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f007d28be6d13fa404016f8f0dc19e8e with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 6ece9b713f64fb969206a1fc295d56b5 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4bbcad05-6c57-4f15-935e-18a7352296ee_characterizeImage_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4bbcad05-6c57-4f15-935e-18a7352296ee_characterizeImage_1039770_3.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54310d8e-7f5d-4b0d-bb35-8d5236fe603e_isr_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54310d8e-7f5d-4b0d-bb35-8d5236fe603e_isr_1039770_3.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f6258e7107abae07db4a8fc8abadd83 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash bd3be456fa2e776196be2a9fa36ae650 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/3a75ba85-39ce-4f18-ae7f-714f163c973f_characterizeImage_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/3a75ba85-39ce-4f18-ae7f-714f163c973f_characterizeImage_1039770_21.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c85f298f-1482-4f8a-adc2-346f466d1423_isr_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c85f298f-1482-4f8a-adc2-346f466d1423_isr_1039770_21.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dade2794deee2925defadeb0e0c55dfb with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash c5865b67eb47de0b7ab83b93ad12724b parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5765792d-4e84-44f2-beb3-3e97f20cc54d_characterizeImage_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5765792d-4e84-44f2-beb3-3e97f20cc54d_characterizeImage_1039673_7.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/46a1dd71-ada5-46dd-8f2e-cab53696e9a2_isr_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/46a1dd71-ada5-46dd-8f2e-cab53696e9a2_isr_1039673_7.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a6cf78ed8ed009787317b97e6c40b00 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 9406cce65eef05d3c6bb7548b0789eb2 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f779bcd-99e6-4242-93eb-2eb209fa1613_characterizeImage_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f779bcd-99e6-4242-93eb-2eb209fa1613_characterizeImage_1039673_32.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/83cb17c2-9ea1-449c-94cd-be991290b3a4_isr_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/83cb17c2-9ea1-449c-94cd-be991290b3a4_isr_1039673_32.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9a79f2fb6e7da2a4e3517deecbd9402 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 8b9201217981896a046e39caf81eeff8 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/6812b309-2544-4edd-8d27-89b212b280ab_characterizeImage_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/6812b309-2544-4edd-8d27-89b212b280ab_characterizeImage_1039769_28.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5974d213-e1c5-4ea1-937c-6d59c2746b81_isr_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5974d213-e1c5-4ea1-937c-6d59c2746b81_isr_1039769_28.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41cb0287f12424c1dfe43cd10903601 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 10eda5fd34e16b8cf019160f88e52a48 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6321dd04-3729-4750-ad17-ac2b34dc0ec9_characterizeImage_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6321dd04-3729-4750-ad17-ac2b34dc0ec9_characterizeImage_1039672_27.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/578203ac-36df-4376-840f-bb5be383ccdd_isr_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/578203ac-36df-4376-840f-bb5be383ccdd_isr_1039672_27.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa4aaf4b21fdf1a3ed8a9bef2abf22b with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 4e9c53f8a63a46903ffa3b9afcbe319a parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/00fabb0a-c111-43e9-a08d-688816824f51_characterizeImage_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/00fabb0a-c111-43e9-a08d-688816824f51_characterizeImage_1039674_12.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/73f5196f-4a9e-4902-a5a9-d9e49603f895_isr_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/73f5196f-4a9e-4902-a5a9-d9e49603f895_isr_1039674_12.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34f1401c54c008085e7033db006a63b7 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash dd31142655d74b578e2fc77361c6398e parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/868aa207-d264-4ba2-b7e3-528e0d401fda_characterizeImage_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/868aa207-d264-4ba2-b7e3-528e0d401fda_characterizeImage_1039770_10.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7babba24-191f-41bb-a684-ee26f89dfaad_isr_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7babba24-191f-41bb-a684-ee26f89dfaad_isr_1039770_10.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05b5c437e88827c0711f1828fd871077 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 62e21b7b18d6b60b201116552a063a76 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/e4305031-2f3f-4162-b91e-fbdc1a461c09_characterizeImage_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/e4305031-2f3f-4162-b91e-fbdc1a461c09_characterizeImage_1039673_18.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e2955ecf-c11c-4941-ad73-2b616a359f40_isr_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e2955ecf-c11c-4941-ad73-2b616a359f40_isr_1039673_18.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab558705867af04136b5ba3c61883498 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 7e911868c1b0b8641aaad68a1e0d3ab4 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/f85fbb4a-f03d-4a83-b664-a7abb06661cd_characterizeImage_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/f85fbb4a-f03d-4a83-b664-a7abb06661cd_characterizeImage_1039770_29.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2fca352e-4058-4123-b056-adba8691987f_isr_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2fca352e-4058-4123-b056-adba8691987f_isr_1039770_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:34 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718fc2d456b860370845f34e93edc9af with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash db11ee834e25bc47c8ac80d0fa5ad379 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/26ee19f5-d188-4385-a275-5cc0c6558369_characterizeImage_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/26ee19f5-d188-4385-a275-5cc0c6558369_characterizeImage_1039674_36.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd78fe2d-2dee-49a0-aafc-75905dced438_isr_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd78fe2d-2dee-49a0-aafc-75905dced438_isr_1039674_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:35 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:37 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:37 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:38 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:38 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 248406f487454e947a602cd6dd503d4a with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 55301090325756c7e7a35e93c7a4047b parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/093fd1c6-6746-4170-bc5e-5ffad1404048_characterizeImage_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/093fd1c6-6746-4170-bc5e-5ffad1404048_characterizeImage_1039768_43.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/60369cd3-c064-4019-9217-5735c3202f90_isr_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/60369cd3-c064-4019-9217-5735c3202f90_isr_1039768_43.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 831c6deb0f5053dc52e6aa801b7a18f1 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 11b082596c6c01c0cb30b2d3b320ebbe parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f4902d84-db08-4ebc-8066-036d394cd089_characterizeImage_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f4902d84-db08-4ebc-8066-036d394cd089_characterizeImage_1039768_8.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/17e7e44a-9b1e-4075-b307-f9ee8f2754ca_isr_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/17e7e44a-9b1e-4075-b307-f9ee8f2754ca_isr_1039768_8.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96bb82cccf1ef1f62a3b94b385de904d with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 8fbfdd904d89c3885c934bd88859b3bd parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f6eeff25-5c9a-42ec-b748-780ce48cdb8d_characterizeImage_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f6eeff25-5c9a-42ec-b748-780ce48cdb8d_characterizeImage_1039674_22.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e11a28c7-0f3f-40b5-93a7-fb18611cf290_isr_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e11a28c7-0f3f-40b5-93a7-fb18611cf290_isr_1039674_22.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07069c4c59345984efdbbc534c95a567 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash ce29401fdbd13439aa89e40b7aa123b1 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/971d3828-ee6e-460d-bc6f-465afbdb30fa_characterizeImage_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/971d3828-ee6e-460d-bc6f-465afbdb30fa_characterizeImage_1039674_4.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e00914e3-7553-40d7-af1b-eb57d387c995_isr_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e00914e3-7553-40d7-af1b-eb57d387c995_isr_1039674_4.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a92b78f4c2cf86a6466662818ac32ce0 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 493d94166c5d5aaa8a7bab47bb73dfa9 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cabc34b9-c051-4b64-bb99-9ba3768d35dc_characterizeImage_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cabc34b9-c051-4b64-bb99-9ba3768d35dc_characterizeImage_1039768_42.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/853ba2a1-5218-4980-89c1-3057aedf7c69_isr_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/853ba2a1-5218-4980-89c1-3057aedf7c69_isr_1039768_42.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96b1d38f1e06ebf8b6324c2b5fdb6a1 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 8cea29e4b9f05c1d6e38427df2d263d8 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/258a6030-a68f-4c16-9e69-1e33ca2ae532_characterizeImage_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/258a6030-a68f-4c16-9e69-1e33ca2ae532_characterizeImage_1039672_56.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/fb29da83-871a-43c5-a456-4b39916e1aff_isr_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/fb29da83-871a-43c5-a456-4b39916e1aff_isr_1039672_56.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53142ca8ae75ceebac802193c9defcd0 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 4051cb6367121b018d1c84328157e029 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/20cdff64-475f-4928-80d8-1c295aa9e620_characterizeImage_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/20cdff64-475f-4928-80d8-1c295aa9e620_characterizeImage_1039768_44.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4500de69-44fa-4e13-9786-85d23bb970c1_isr_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4500de69-44fa-4e13-9786-85d23bb970c1_isr_1039768_44.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e30ef5fae2aeca6c1a563a0567e8c77a with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 58155b3385cb6eb75a3a588bb1574178 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9114e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3606cae8-38ad-4f71-91b2-713a42f8204d_characterizeImage_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3606cae8-38ad-4f71-91b2-713a42f8204d_characterizeImage_1039673_35.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/b2a23c03-0105-4f28-89be-cf7e58e547f3_isr_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/b2a23c03-0105-4f28-89be-cf7e58e547f3_isr_1039673_35.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db97e49255abb819d3dd5f9c23049568 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 359f990620a15874d1a3c79635d427d6 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9111c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/f65a691b-ddf5-439c-b8a0-49d9ed910e09_characterizeImage_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/f65a691b-ddf5-439c-b8a0-49d9ed910e09_characterizeImage_1039673_17.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e0303cec-e056-41ec-b64b-09660376a9c0_isr_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/e0303cec-e056-41ec-b64b-09660376a9c0_isr_1039673_17.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21f474b4482c741141a9281e5ff35824 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash d66f95d879168e045612cd095d8480cc parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/68bd8e22-1d72-4b75-bda9-05656a0ec910_characterizeImage_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/68bd8e22-1d72-4b75-bda9-05656a0ec910_characterizeImage_1039768_3.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61c2298f-9016-45dc-9002-206cccdff9f9_isr_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61c2298f-9016-45dc-9002-206cccdff9f9_isr_1039768_3.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25346798229f3b151c564c58e0f33af with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 563b07ec010a4e758c02e2f730bef9e3 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/71e01d29-3314-47d9-83a6-66d8b10a8cec_characterizeImage_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/71e01d29-3314-47d9-83a6-66d8b10a8cec_characterizeImage_1039770_33.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/06f1fa75-6f8e-4369-8b00-1fd154903d9c_isr_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/06f1fa75-6f8e-4369-8b00-1fd154903d9c_isr_1039770_33.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b5dc5d3a31c5f58fa9334592cffdf8c with result from task 141 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:38 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 8f3d873df612e3ca3454eaf6c475cde4 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d5b29802-39f6-4953-9c3c-9cbf51cb3b96_characterizeImage_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d5b29802-39f6-4953-9c3c-9cbf51cb3b96_characterizeImage_1039768_62.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/24ea5d9c-e07d-497d-8e21-e867bf8bd4f8_isr_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/24ea5d9c-e07d-497d-8e21-e867bf8bd4f8_isr_1039768_62.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3359f6554f6ca5afa66fea6d084a29d9 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 1f5afdc194dd7d501775803908285256 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1b1306-f27a-4617-a12a-1e25f9a575a0_characterizeImage_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1b1306-f27a-4617-a12a-1e25f9a575a0_characterizeImage_1039673_22.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0fabdd96-12f5-43ad-9569-d6fe067a7369_isr_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0fabdd96-12f5-43ad-9569-d6fe067a7369_isr_1039673_22.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c49d1ed29fe66627b2cb58cd0ab111 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 02c6221caecc44ccbf086fcc37ae2c95 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce86a028-76b9-4b1b-b03c-bfaad165fa31_characterizeImage_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce86a028-76b9-4b1b-b03c-bfaad165fa31_characterizeImage_1039770_58.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/156f9a6f-f7c1-4d45-a5fb-672b3f56069a_isr_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/156f9a6f-f7c1-4d45-a5fb-672b3f56069a_isr_1039770_58.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48a13e202ea0c04d04799dc11a9ba864 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 5ad524bd5e01baf02f36487dabbfe36f parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/0fff7db2-dd27-48d5-a5b5-380208a6c8c2_characterizeImage_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/0fff7db2-dd27-48d5-a5b5-380208a6c8c2_characterizeImage_1039673_36.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d1a9adfc-88db-49d6-b46c-8ae5ffe761ff_isr_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d1a9adfc-88db-49d6-b46c-8ae5ffe761ff_isr_1039673_36.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b991c484daf27d6cc83bdf26d8292361 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash da8b53625ca5a1298070775e852fb5a4 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fec686f0-a0c1-4b03-945c-5eeb600b4575_characterizeImage_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fec686f0-a0c1-4b03-945c-5eeb600b4575_characterizeImage_1039672_58.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7cc3f24e-88dc-4c51-a0f5-e565856a2e35_isr_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7cc3f24e-88dc-4c51-a0f5-e565856a2e35_isr_1039672_58.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91b47a54f6676d279befd1b02536b75e with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash afc66dc8d4e4640e097a6bd08728dcbc parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/81ad6a0a-6746-49ed-b0f5-6cb5d47319fc_characterizeImage_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/81ad6a0a-6746-49ed-b0f5-6cb5d47319fc_characterizeImage_1039672_13.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/1f7256f7-d2a1-4164-83b4-e1468a1055a9_isr_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/1f7256f7-d2a1-4164-83b4-e1468a1055a9_isr_1039672_13.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce729593a644f904b0ae6805831db6f with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash bb3d97b79721b6d83b47a45f77e7cd64 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4dafcc86-0c02-470c-b79a-c31aa58cf463_characterizeImage_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4dafcc86-0c02-470c-b79a-c31aa58cf463_characterizeImage_1039672_40.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/df76f3a7-3859-40b2-9f02-d7516a9f931d_isr_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/df76f3a7-3859-40b2-9f02-d7516a9f931d_isr_1039672_40.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478cdd4c9930286b2af9670ea30133e7 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash ae23b3f1043a4182d26a9ef301f40f3e parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/1099bcda-4c7a-4f05-8023-36e937398db3_characterizeImage_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/1099bcda-4c7a-4f05-8023-36e937398db3_characterizeImage_1039768_50.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/fa07acac-a19c-4927-90e4-23dbab897f5d_isr_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/fa07acac-a19c-4927-90e4-23dbab897f5d_isr_1039768_50.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0591229b2c819611c83e4e7fbce0b40 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash cf629c92a3e90ffc48af473dbd8cb54f parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93787240-5f77-4d2e-8f7b-4bea957c3da1_characterizeImage_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93787240-5f77-4d2e-8f7b-4bea957c3da1_characterizeImage_1039674_40.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e6d71bd1-09db-4fae-93ca-d644a6764343_isr_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/e6d71bd1-09db-4fae-93ca-d644a6764343_isr_1039674_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:38 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:38 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:39 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:39 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:39 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:39 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:39 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7c440db18cd4ab325e9a06a431be67 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 15ce1ba5a16f40a918c5a8172d5949ff parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/732e6f15-f442-4ba2-aec9-faf666558a36_characterizeImage_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/732e6f15-f442-4ba2-aec9-faf666558a36_characterizeImage_1039768_16.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4d609123-234a-4ff4-b86a-fc2b519609b8_isr_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/4d609123-234a-4ff4-b86a-fc2b519609b8_isr_1039768_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:40 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f173fccb38cae1380fef1f76ee4683fc with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash c13f72f8baeda394d33fd05082ee6269 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7693f361-b390-41a0-a43d-fdb3beab453d_characterizeImage_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7693f361-b390-41a0-a43d-fdb3beab453d_characterizeImage_1039673_48.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1533ebea-7697-4043-ab14-c2fc22ff3746_isr_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1533ebea-7697-4043-ab14-c2fc22ff3746_isr_1039673_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:40 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:40 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:40 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:41 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 606a2ebcb2d27441dabdaa54e524213f with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 4a2fa50a69d1fe80374a884dcc4b6c2d parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8da4a513-2333-4b16-b821-0900119cfa13_characterizeImage_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8da4a513-2333-4b16-b821-0900119cfa13_characterizeImage_1039674_48.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88f4c182-6fbc-4a33-9bc5-390d3c09a78b_isr_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88f4c182-6fbc-4a33-9bc5-390d3c09a78b_isr_1039674_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:41 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edceedc3513fa5698ba2afb2fe57aa98 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash a973d902d7ed6a5f308b69dee725cea4 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d8fa4c85-c971-4ae9-8367-0fb8a6bbd6dd_characterizeImage_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d8fa4c85-c971-4ae9-8367-0fb8a6bbd6dd_characterizeImage_1039768_59.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/219a5b0f-4cc4-4b39-9b7e-0ec131e2279f_isr_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/219a5b0f-4cc4-4b39-9b7e-0ec131e2279f_isr_1039768_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:41 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:42 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:43 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:43 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:43 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:44 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:44 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:45 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:45 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1dfbd44b9eaeb5d3d6da9ea44f91d9a3 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash f70ed9c5ae6c1e0eb316832788aa5ffc parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fb85db6-b53f-4852-be7e-16daf753d118_characterizeImage_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fb85db6-b53f-4852-be7e-16daf753d118_characterizeImage_1039672_46.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ac4e3a50-5e26-4f2b-b735-b9eac6ab5b83_isr_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ac4e3a50-5e26-4f2b-b735-b9eac6ab5b83_isr_1039672_46.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc1bd46ad69d2bf0861a86540ac0776d with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 52f416e0ba15f220c025f3cece47f710 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a2143ae2-2d97-4f9f-80dc-1dbc08fed8ca_characterizeImage_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a2143ae2-2d97-4f9f-80dc-1dbc08fed8ca_characterizeImage_1039769_36.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/eeeeda02-fe8e-44e7-9b49-0a55d72248f0_isr_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/eeeeda02-fe8e-44e7-9b49-0a55d72248f0_isr_1039769_36.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86f26aa24ad24f70260d41b82a1693ed with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 510d1e4f21edea0299d2651cc9758359 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/6e881d82-d261-4f05-a890-e07ff6895109_characterizeImage_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/6e881d82-d261-4f05-a890-e07ff6895109_characterizeImage_1039768_14.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/493fbd14-5c1b-4bf5-a6b7-7b0981879060_isr_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/493fbd14-5c1b-4bf5-a6b7-7b0981879060_isr_1039768_14.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac6df29ccda08db67e104c43ed8e8c5d with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 17a468ec19383d49f8fd0bd63a3385bb parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8afb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/097c9347-50d8-45e1-9f00-d8fcb2dfd968_characterizeImage_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/097c9347-50d8-45e1-9f00-d8fcb2dfd968_characterizeImage_1039768_53.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61b4bf9d-3f64-4250-a316-380d3cf135a9_isr_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/61b4bf9d-3f64-4250-a316-380d3cf135a9_isr_1039768_53.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c6b4bc249a4138ade4297aa3bab1462 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 6bbad86bb971fcd66f0d678aae5f1406 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/eff8265a-bbb0-4ac1-9602-314ad50fb2e1_characterizeImage_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/eff8265a-bbb0-4ac1-9602-314ad50fb2e1_characterizeImage_1039674_37.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b7111778-4794-4823-980b-b35fd3791f0a_isr_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b7111778-4794-4823-980b-b35fd3791f0a_isr_1039674_37.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1d3558bbde63c1488183b897bcd1b2 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 0c24eb0c477817115b3a28e0490adf3e parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fa9b59ba-625d-4507-90af-4452ee1a0dc9_characterizeImage_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fa9b59ba-625d-4507-90af-4452ee1a0dc9_characterizeImage_1039672_15.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64e33a9b-007e-4598-8ff1-bc49099f1361_isr_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64e33a9b-007e-4598-8ff1-bc49099f1361_isr_1039672_15.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e2fe72fdf9c14101b5c4bf0a69ff4ed with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash b3d11955308e52b7aba778c6660b2bac parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/d32e2410-a74c-4808-8c58-72af9318f3cb_characterizeImage_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/d32e2410-a74c-4808-8c58-72af9318f3cb_characterizeImage_1039674_21.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f2154792-5301-408c-acb3-24fc5b768bb6_isr_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f2154792-5301-408c-acb3-24fc5b768bb6_isr_1039674_21.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7727d7cf4351310ebc4701ca1f08646 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash c5ca329cb46a76bceabd61c80cac9af9 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/fb8f627f-8a6a-4b28-871a-69d03d20b186_characterizeImage_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/fb8f627f-8a6a-4b28-871a-69d03d20b186_characterizeImage_1039674_6.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d0811059-7b39-4c12-a485-7d6078099f7a_isr_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d0811059-7b39-4c12-a485-7d6078099f7a_isr_1039674_6.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry feceb4d01392fb638d2f7a0b8116282b with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 493b4d8f27ae8d7d8178fed41bfa0c52 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/677f622b-cb3c-4b4c-99e6-0df3489639cb_characterizeImage_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/677f622b-cb3c-4b4c-99e6-0df3489639cb_characterizeImage_1039769_34.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bef9535d-8989-47a6-96e7-f54cb8d232fc_isr_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bef9535d-8989-47a6-96e7-f54cb8d232fc_isr_1039769_34.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02077c11b45767be238d027da728263c with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 45a39ee549df54d7d2ad20e8ed31a775 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1c7568aa-efb7-4dab-846e-7ca56a69bbb3_characterizeImage_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1c7568aa-efb7-4dab-846e-7ca56a69bbb3_characterizeImage_1039770_36.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e9f6fe89-8534-482e-824d-2a84949016c2_isr_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e9f6fe89-8534-482e-824d-2a84949016c2_isr_1039770_36.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc6439d73d3d6547de9a5f893ade0d13 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 16e5d60e6958d0e1a5d67f65d8f5d76b parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/710c25d2-fc89-4ebb-b50e-fea4eb21713d_characterizeImage_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/710c25d2-fc89-4ebb-b50e-fea4eb21713d_characterizeImage_1039770_20.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0d89c77c-f4d3-44b5-9a1f-19e8f6104a88_isr_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0d89c77c-f4d3-44b5-9a1f-19e8f6104a88_isr_1039770_20.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ded64f79a0f7059af2f47326eb210d86 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 968b9381372f66dfc80d0b1970fa9629 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3435543c-793c-4c7f-adaa-03d4d0ca2690_characterizeImage_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3435543c-793c-4c7f-adaa-03d4d0ca2690_characterizeImage_1039673_27.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c70fef74-952c-45aa-8432-a70e66311aaf_isr_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c70fef74-952c-45aa-8432-a70e66311aaf_isr_1039673_27.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 344070072464d596f1191530810cb7e4 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash f0b8691f23694d78e4c798e5801734b7 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8afe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c75243c2-ecf9-4dce-9b40-8cc74d2abaed_characterizeImage_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c75243c2-ecf9-4dce-9b40-8cc74d2abaed_characterizeImage_1039673_25.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/45dce31a-346d-4a8f-ac1d-2b0bb210a3ba_isr_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/45dce31a-346d-4a8f-ac1d-2b0bb210a3ba_isr_1039673_25.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ab158735373fa6d049757f4b1f23c36 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 0de2bd72f1f4106cfdf2dc4e41fe0494 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4e0fdea0-6fa0-42db-8d23-d9bcc569384a_characterizeImage_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4e0fdea0-6fa0-42db-8d23-d9bcc569384a_characterizeImage_1039768_55.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/644a1b94-d832-42ed-8afe-cdc09c8b60a8_isr_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/644a1b94-d832-42ed-8afe-cdc09c8b60a8_isr_1039768_55.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ec037821d46e12ce624cdc17a7078a9 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 5fcde757426a640d131faff0fb648127 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5b2b4ae-5298-45c5-8396-96c9c762ac44_characterizeImage_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5b2b4ae-5298-45c5-8396-96c9c762ac44_characterizeImage_1039672_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c9ab1468-45b5-4cff-b40b-f47f62dc5efc_isr_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c9ab1468-45b5-4cff-b40b-f47f62dc5efc_isr_1039672_49.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d86af8e209f46bf855241f8b0b9a337 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 8c93f446535928e23a9372941d1d1f45 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/845c5fe4-159d-4d2b-9c2a-ad8a493112c1_characterizeImage_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/845c5fe4-159d-4d2b-9c2a-ad8a493112c1_characterizeImage_1039770_35.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f97b9ff7-9767-4e89-8011-ffec7ba1ab8a_isr_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f97b9ff7-9767-4e89-8011-ffec7ba1ab8a_isr_1039770_35.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 513fec5a5fb30c652fa7dd173be87718 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 470b4da64521f297b97f445d215e2895 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/32c7d364-e609-49e3-83aa-6acdcf102579_characterizeImage_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/32c7d364-e609-49e3-83aa-6acdcf102579_characterizeImage_1039672_50.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd2e1811-bac3-418b-807a-27882d5a838c_isr_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd2e1811-bac3-418b-807a-27882d5a838c_isr_1039672_50.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 751a17a75c6fd098e509c4c4d661e0f3 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash e8390a8ffd32022ce99f5eccd03136c0 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f80c6eca-0fed-470f-a287-dcc28e88a634_characterizeImage_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f80c6eca-0fed-470f-a287-dcc28e88a634_characterizeImage_1039768_46.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/766af1f8-a676-417a-abe7-e3d578975c78_isr_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/766af1f8-a676-417a-abe7-e3d578975c78_isr_1039768_46.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dad8f02032272a3e05ad632062f0f659 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash d1ffef71e3498f373260a37e9f28ddb0 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a7f9c228-2da1-499d-a620-4fce8e9e932a_isr_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a7f9c228-2da1-499d-a620-4fce8e9e932a_isr_1039673_2.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebeee5f886c70cf099ddd0c1974203b4 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 4b459818656df5d94200e00bb161d982 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8adda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0be4fa6e-563e-4887-bf17-ed9f520f014b_characterizeImage_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0be4fa6e-563e-4887-bf17-ed9f520f014b_characterizeImage_1039769_24.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ecc8c965-ea6a-4990-b0a2-3457d7c6d363_isr_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ecc8c965-ea6a-4990-b0a2-3457d7c6d363_isr_1039769_24.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd74cd1f96967008ad302fa84398d2bd with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 7ccd8edfc6f2af0d3bbb638859c4a105 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6a98c9f3-d213-494a-aa1d-b623561b8e6a_characterizeImage_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6a98c9f3-d213-494a-aa1d-b623561b8e6a_characterizeImage_1039674_23.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1ec1c172-7e5f-45c1-a410-c9a00a015a12_isr_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1ec1c172-7e5f-45c1-a410-c9a00a015a12_isr_1039674_23.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca3d17a246bf32ab7b2b32c95d27550 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 34efcbbced1d463f7ef6b1d5dafcd96a parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/44673071-e92a-409d-96cc-7481821f1efb_characterizeImage_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/44673071-e92a-409d-96cc-7481821f1efb_characterizeImage_1039768_32.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c76b0826-198a-4bf2-bfa6-bbceeca4587e_isr_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c76b0826-198a-4bf2-bfa6-bbceeca4587e_isr_1039768_32.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68645d6a96d2dfd92c56395050e4769 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 9318c5445167ff0f489b9a6f478efd29 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4eebf4fc-cfe5-49c4-8b9d-010f33326ff9_characterizeImage_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4eebf4fc-cfe5-49c4-8b9d-010f33326ff9_characterizeImage_1039674_24.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/96a9f206-3717-44ed-a79f-ffc388a39ca8_isr_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/96a9f206-3717-44ed-a79f-ffc388a39ca8_isr_1039674_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:46 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:47 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:47 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:47 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:47 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d09c431c1b53f9035a2d0503bc797ae4 with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash dd7fff069a399ccf118b16881342b2a7 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/078297e9-914b-42b8-a55c-79908846fe02_characterizeImage_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/078297e9-914b-42b8-a55c-79908846fe02_characterizeImage_1039673_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f8c22207-4fad-442a-888e-42905ffa35f3_isr_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f8c22207-4fad-442a-888e-42905ffa35f3_isr_1039673_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:47 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c557038d2a50d119281a7d269e3e6e3e with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash c9e7bbe3174a85ea919212a0b7bf3e22 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cf228451-63ac-42ee-900d-d1ec0b3717a6_characterizeImage_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cf228451-63ac-42ee-900d-d1ec0b3717a6_characterizeImage_1039770_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/46927fca-bb27-4eba-af2d-c681b455818f_isr_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/46927fca-bb27-4eba-af2d-c681b455818f_isr_1039770_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30945af285df5081cf2d6aaeaa33370e with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 100878fa02f63f4dd175ec8feed28fa4 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1328c4b9-8025-4a97-bb05-202a006c45c5_characterizeImage_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1328c4b9-8025-4a97-bb05-202a006c45c5_characterizeImage_1039672_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8047c5b7-e005-416f-9993-0ec658ab944f_isr_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8047c5b7-e005-416f-9993-0ec658ab944f_isr_1039672_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:38:48 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a74cc29f10f89da1b0520739b550d92a with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash d7f0153b0c0e35a26658de17d21d8d49 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/94c7ab4b-6f6a-43d1-a27d-e207a5111230_characterizeImage_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/94c7ab4b-6f6a-43d1-a27d-e207a5111230_characterizeImage_1039673_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9ca6fcce-72d1-4f55-aef0-a3165ee14cdd_isr_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9ca6fcce-72d1-4f55-aef0-a3165ee14cdd_isr_1039673_51.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f76bc86b34a9b0d9f5c5f5267438237 with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash 2603df5a9fb03d781d9512f36f054bd4 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2167a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8c9bb852-e3db-49d4-8e2b-48649d79968a_characterizeImage_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8c9bb852-e3db-49d4-8e2b-48649d79968a_characterizeImage_1039674_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/2c407beb-96cb-47eb-9f50-27f15e813017_isr_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/2c407beb-96cb-47eb-9f50-27f15e813017_isr_1039674_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 803cd429672e2391732863a1cd4a2e51 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash 9eec5ed102f9f1c0d35c22c88d1c446a parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/dca5480b-6f8d-4231-bcf9-236d81d146f5_characterizeImage_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/dca5480b-6f8d-4231-bcf9-236d81d146f5_characterizeImage_1039673_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4f487394-45c1-4679-a8c6-cc30adea4c00_isr_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4f487394-45c1-4679-a8c6-cc30adea4c00_isr_1039673_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeaa5df4365c502b38d9b7ca4e88443c with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash aab0ff9d9d8af1da7424bdcef98bbdc7 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8034ac87-5ede-423e-adc4-0196cb1bb030_characterizeImage_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8034ac87-5ede-423e-adc4-0196cb1bb030_characterizeImage_1039768_25.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1ba94237-d74f-4086-8584-cc23eeae4b89_isr_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1ba94237-d74f-4086-8584-cc23eeae4b89_isr_1039768_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:49 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f71e462e30134014546d6202e50364eb with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 816d12c01bb7d9de22c13b399af9a91a parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/37908053-dba9-4815-a624-c31c3c8cd487_characterizeImage_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/37908053-dba9-4815-a624-c31c3c8cd487_characterizeImage_1039770_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54a2c220-4834-4ed1-9aeb-d57a247c8908_isr_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/54a2c220-4834-4ed1-9aeb-d57a247c8908_isr_1039770_1.stderr parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e30246062ae8f79aa996a5a40e1b30d4 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash e4658391f3241a4dc6328941d74605e5 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/353b793a-8fea-43ae-8249-b45dc8d0bbc3_characterizeImage_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/353b793a-8fea-43ae-8249-b45dc8d0bbc3_characterizeImage_1039674_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/521b2b1a-2ce3-4b8b-8f5b-4c5e9519fb08_isr_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/521b2b1a-2ce3-4b8b-8f5b-4c5e9519fb08_isr_1039674_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:50 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87b1ffe450602789016fa29356657359 with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 715ee1f2861304f696ba37067208c2fd parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/197f2221-9f27-42a5-999d-f7732dce25c7_characterizeImage_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/197f2221-9f27-42a5-999d-f7732dce25c7_characterizeImage_1039673_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/40e9e1ff-8fb4-40eb-98b7-989728bc204b_isr_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/40e9e1ff-8fb4-40eb-98b7-989728bc204b_isr_1039673_39.stderr parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69287b1c2d65087b25db94dee4ebdb8e with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash df680e162b448748e0c5577671df6012 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9a3aa1c6-f5bc-4f29-bfc6-ac799b72320b_characterizeImage_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9a3aa1c6-f5bc-4f29-bfc6-ac799b72320b_characterizeImage_1039673_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d8e906a-a863-4399-8a25-789d20afa2e2_isr_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d8e906a-a863-4399-8a25-789d20afa2e2_isr_1039673_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:50 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 023d0b8f0224792ed5d3f8991a2c586d with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash bf6807fa6473ab24ef678a83236d8746 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/27202d2c-a53b-47e9-a409-133f05cd4d2a_characterizeImage_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/27202d2c-a53b-47e9-a409-133f05cd4d2a_characterizeImage_1039770_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3922ca5f-525c-4295-8be9-8cfa91224afb_isr_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3922ca5f-525c-4295-8be9-8cfa91224afb_isr_1039770_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:50 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:50 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e192be1e180bc0806f2efd0719064d6d with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 07bfbbb997e110b35b37ddae099dabc8 parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0744a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/48c4d3ae-e724-4b12-a63f-75a9ad973c6e_characterizeImage_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/48c4d3ae-e724-4b12-a63f-75a9ad973c6e_characterizeImage_1039770_18.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0e90c774-8304-4d33-b735-607eb84cbe24_isr_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0e90c774-8304-4d33-b735-607eb84cbe24_isr_1039770_18.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a96b202703327dbee2db7ccf460fee2a with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 987b495a410b95b04335e84669d967e2 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/de9b8ff7-556c-4f78-bb3c-33326ad64e65_characterizeImage_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/de9b8ff7-556c-4f78-bb3c-33326ad64e65_characterizeImage_1039674_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8d6d226f-3e70-4631-a521-cf4d0574460e_isr_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8d6d226f-3e70-4631-a521-cf4d0574460e_isr_1039674_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efb99b472b3d9f4a2ea9ffa4f32a5f57 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 4b2dc2aeb056a91953a6d5d7262e7d45 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/512664e1-6b53-474a-8719-374c16b7e5f2_characterizeImage_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/512664e1-6b53-474a-8719-374c16b7e5f2_characterizeImage_1039674_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/952614bc-0142-4017-aecb-e8222e0a7eca_isr_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/952614bc-0142-4017-aecb-e8222e0a7eca_isr_1039674_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc4f5c7e6845f02048f7bceb6acee210 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash db33117a60b0c6f5a082856a5ece9eaf parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0747c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8de2d7a6-a957-46c3-8026-5e45534a3e9b_characterizeImage_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8de2d7a6-a957-46c3-8026-5e45534a3e9b_characterizeImage_1039768_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/72a0ad8b-c54f-4363-80c8-217102c29c9f_isr_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/72a0ad8b-c54f-4363-80c8-217102c29c9f_isr_1039768_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:51 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c998ceac0a5b2b077643b0b6316322f with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 67cf333fe76cbafda1aa0e166f350a12 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9102c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/261b5cbb-43ee-4b43-9694-937411f2c2ef_characterizeImage_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/261b5cbb-43ee-4b43-9694-937411f2c2ef_characterizeImage_1039769_59.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8fbe1e8a-37af-472f-b369-93afd442ef88_isr_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8fbe1e8a-37af-472f-b369-93afd442ef88_isr_1039769_59.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11c184ed1a7dd298471915c23ae9b8cb with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 6bb640b13808bd383b94369b5f24dc73 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/77e4b5ac-32c4-42c3-bed8-b3f9eeef0d20_characterizeImage_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/77e4b5ac-32c4-42c3-bed8-b3f9eeef0d20_characterizeImage_1039769_48.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/938e6cbf-c9b0-4091-889f-d0ab443f6c4e_isr_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/938e6cbf-c9b0-4091-889f-d0ab443f6c4e_isr_1039769_48.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96fddbafe60444384c38e1a80115ee30 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash cd4c5657636372c8a95f5e52ee33a2a5 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2498c243-035e-4db5-9b53-c851205d774b_characterizeImage_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2498c243-035e-4db5-9b53-c851205d774b_characterizeImage_1039770_11.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f933dd07-1046-47af-900e-17d8a0670342_isr_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f933dd07-1046-47af-900e-17d8a0670342_isr_1039770_11.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 159a5fbbeba39849386e5409d73a0f3a with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 38681c4aa6502d0032dba0146001b71d parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f00c1689-38b3-4cde-a5f6-5eb17c304435_characterizeImage_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f00c1689-38b3-4cde-a5f6-5eb17c304435_characterizeImage_1039674_35.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c11b1c75-b7d6-458d-829b-05a01ed6546d_isr_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c11b1c75-b7d6-458d-829b-05a01ed6546d_isr_1039674_35.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 183abc6c0ea31b786b907a46a138c9cf with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 2e1bbd85fe3237163684b74880242d0b parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/15b7d610-1bc4-4139-9e71-f4e0ba063228_characterizeImage_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/15b7d610-1bc4-4139-9e71-f4e0ba063228_characterizeImage_1039770_32.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/87a46004-8530-400a-9779-2d4da889b0ba_isr_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/87a46004-8530-400a-9779-2d4da889b0ba_isr_1039770_32.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79964494b3bf2e79e86bf5dd014ccd13 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash e0393df87e3f9812cc03e6bea3e533ab parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8751ab07-8308-4ddb-8096-d7274e7ec00c_characterizeImage_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8751ab07-8308-4ddb-8096-d7274e7ec00c_characterizeImage_1039673_6.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/66a7162a-fc14-44e6-a8c6-191cac61e9f1_isr_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/66a7162a-fc14-44e6-a8c6-191cac61e9f1_isr_1039673_6.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f61cda82eb2ef95a0b1633b8f771b12 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 70b4ff97275aebd0a837709f93705c22 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a6a98c36-9bcd-4a5c-905c-98049b8b3dfd_characterizeImage_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a6a98c36-9bcd-4a5c-905c-98049b8b3dfd_characterizeImage_1039674_1.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/68261430-ccc3-4923-a72c-d6003d9bb7d6_isr_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/68261430-ccc3-4923-a72c-d6003d9bb7d6_isr_1039674_1.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a029ea56282d126c21af5c8b38b9240f with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash a16e724587f18841d031d3c411c21440 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1f288f1d-a4ba-4dac-b3b7-f033ea5c1d7b_characterizeImage_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1f288f1d-a4ba-4dac-b3b7-f033ea5c1d7b_characterizeImage_1039672_39.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a0de1073-3699-47d1-a7ec-458ba7968a62_isr_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a0de1073-3699-47d1-a7ec-458ba7968a62_isr_1039672_39.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c61978231f756f9ca30f09b48974d416 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 40324ee4746dc9625d0e4f5980058123 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9c51f72f-3148-402e-952c-a7b4bf4bf7a1_characterizeImage_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9c51f72f-3148-402e-952c-a7b4bf4bf7a1_characterizeImage_1039770_41.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/843b5d50-fae9-4f78-afbc-44cd046d4d33_isr_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/843b5d50-fae9-4f78-afbc-44cd046d4d33_isr_1039770_41.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6207575b3031cffc214bad9e32fb0bb1 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash a2dd4a99ad20326682d8fa00a9b91e81 parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/8d12935f-9b67-4008-b81a-32354a87f827_characterizeImage_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/8d12935f-9b67-4008-b81a-32354a87f827_characterizeImage_1039770_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/eeb43dbd-8191-44e0-87fe-75daca31486e_isr_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/eeb43dbd-8191-44e0-87fe-75daca31486e_isr_1039770_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:52 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6761811f4c80397193bb84b7eac781ad with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash bca534d94a3611ae378e185604473b94 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5510bb8-bea2-4be8-b3d2-04aafdfc1e05_characterizeImage_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5510bb8-bea2-4be8-b3d2-04aafdfc1e05_characterizeImage_1039672_23.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/2e80db22-f184-4f12-9ac3-23234b174952_isr_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/2e80db22-f184-4f12-9ac3-23234b174952_isr_1039672_23.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3af703200145f3b9610ecb86d57e3df with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 855b723644171f6109bea49da4ebbbb6 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f812cbce-8e10-4b56-934b-5e46296e53c5_characterizeImage_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f812cbce-8e10-4b56-934b-5e46296e53c5_characterizeImage_1039769_46.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/cccf163c-09e4-43d2-ae03-c7a110cb82fd_isr_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/cccf163c-09e4-43d2-ae03-c7a110cb82fd_isr_1039769_46.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da284928af5c997ec0622433800475ca with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 55a512d336e5f144e092de75cf2f2a8c parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a0eba6d8-10f4-4672-948d-9ac45821e746_characterizeImage_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a0eba6d8-10f4-4672-948d-9ac45821e746_characterizeImage_1039672_42.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/689c73ba-02eb-4175-ae34-749cfb92b638_isr_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/689c73ba-02eb-4175-ae34-749cfb92b638_isr_1039672_42.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae0e5346ead7be3b42a0695a00788e0f with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 182dbd14ac29a617f66f07f37843bfe6 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/97efbac6-69e9-410e-8cf9-fc72389ae604_characterizeImage_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/97efbac6-69e9-410e-8cf9-fc72389ae604_characterizeImage_1039769_14.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/48e9fda6-54c9-461a-abf4-5d79737fd0da_isr_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/48e9fda6-54c9-461a-abf4-5d79737fd0da_isr_1039769_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd2f6a30bed3c60c8f3ce8dc9c0e455c with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 51bfdde2367509a3ad00d8591bd45a1a parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4213b436-7abf-43bd-bf9e-216d5695015e_characterizeImage_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4213b436-7abf-43bd-bf9e-216d5695015e_characterizeImage_1039672_52.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/366bc16f-26a2-41da-9afe-995ce82e830b_isr_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/366bc16f-26a2-41da-9afe-995ce82e830b_isr_1039672_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:53 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 552ded2a5773de002992f35fa829d30f with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 798176a05a1e60b03efb16c5360186a1 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92b5740b-4c21-49bb-9b70-ae17659e09fe_characterizeImage_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92b5740b-4c21-49bb-9b70-ae17659e09fe_characterizeImage_1039672_62.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d75208d5-b812-46a4-9475-5eb261eff473_isr_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d75208d5-b812-46a4-9475-5eb261eff473_isr_1039672_62.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 486f49c76cd5c16a4641cc6dcb8f6bae with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 0bb83e747714d000469f692f5c9dce35 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/fe5ce469-9429-473d-b1c6-0933bb3f1b1d_characterizeImage_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/fe5ce469-9429-473d-b1c6-0933bb3f1b1d_characterizeImage_1039770_30.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1219a93e-d33e-45bc-be25-1c12214bebf2_isr_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1219a93e-d33e-45bc-be25-1c12214bebf2_isr_1039770_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:54 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:54 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:54 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b5e239966e6c90d5d39f5be5f6735fd with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 1efa82da371380349cce8250f27846d7 parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bd105014-1948-4f50-8a6b-a2206178713f_characterizeImage_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bd105014-1948-4f50-8a6b-a2206178713f_characterizeImage_1039673_11.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ae202a23-0f7a-4b3e-9e4e-c78a4883ffa1_isr_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ae202a23-0f7a-4b3e-9e4e-c78a4883ffa1_isr_1039673_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:54 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:55 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:55 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:56 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:56 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:56 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:56 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:56 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caf34593190c5c801ff2e2be34668383 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 49fc0a323b9f85efc2965b7a062877ea parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6f7fcdec-2285-48c8-9816-80787ae1fbce_characterizeImage_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6f7fcdec-2285-48c8-9816-80787ae1fbce_characterizeImage_1039770_34.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/69a34123-57de-4015-bf92-2c6f6c8278ef_isr_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/69a34123-57de-4015-bf92-2c6f6c8278ef_isr_1039770_34.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2391eca6f9b69c53ba2bdcf8f9d8ee4 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 31385ba72d4703e6426c567c5b01b4eb parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f0d1b283-0fb9-4b24-be85-19126f71dbe1_characterizeImage_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f0d1b283-0fb9-4b24-be85-19126f71dbe1_characterizeImage_1039768_13.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04df2adb-1006-4fd0-8623-5efa75494338_isr_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04df2adb-1006-4fd0-8623-5efa75494338_isr_1039768_13.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea56403a6e6f98ed84f129e36b9b9f3 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash ccb0208067ffae947d8d295157999a3a parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e4e3062-a45a-442f-917e-69d2dcd4f6d9_characterizeImage_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e4e3062-a45a-442f-917e-69d2dcd4f6d9_characterizeImage_1039672_9.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/915df0c9-be1c-42ba-8c1a-5be8948223b3_isr_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/915df0c9-be1c-42ba-8c1a-5be8948223b3_isr_1039672_9.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2ab67320f63126642095c4e5f472a15 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 74b828052bcef5a2ff4817e7fec8c3e7 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b11c25a4-0d72-4cf4-8b56-205a70dda6bc_characterizeImage_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b11c25a4-0d72-4cf4-8b56-205a70dda6bc_characterizeImage_1039672_16.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/efd21eab-a7e5-40dd-b0da-c186708f5280_isr_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/efd21eab-a7e5-40dd-b0da-c186708f5280_isr_1039672_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d992974e645a132bf93b6feccdedfc3a with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 36a42f58f0cef3b6c213c04c4f3204e0 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/75cac794-0f22-46e9-aa42-5cadc8f9ddaf_characterizeImage_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/75cac794-0f22-46e9-aa42-5cadc8f9ddaf_characterizeImage_1039674_47.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/35d92771-1fca-4fc9-b94a-b4ed56471356_isr_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/35d92771-1fca-4fc9-b94a-b4ed56471356_isr_1039674_47.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0198a9f42341ee9f0deb8ca8121c3c5c with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash c08c327641201afa55ffa2ce817eb9fd parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a5748f1c-92f4-44f8-8a1d-27914fde4a9e_characterizeImage_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a5748f1c-92f4-44f8-8a1d-27914fde4a9e_characterizeImage_1039674_10.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88294abe-b1de-4925-899c-f6fd62e1d86c_isr_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/88294abe-b1de-4925-899c-f6fd62e1d86c_isr_1039674_10.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8fa6c63621988b107bf0eaf0caf3d22 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 4e9d8455c5e2929846d64989b6fe1336 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a8f51ce1-7a32-4701-9205-beaad917eb01_characterizeImage_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a8f51ce1-7a32-4701-9205-beaad917eb01_characterizeImage_1039769_39.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0eb1d2bb-589b-4ae5-a96d-0fe4721b9282_isr_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0eb1d2bb-589b-4ae5-a96d-0fe4721b9282_isr_1039769_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:57 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:58 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fed7d1f63c1b6a5988f812b30878560 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 3dde4c58aacf6f96d9cbb8bb0f02ae4e parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8ab9f918-e3b3-451a-928b-2abd5fad9fe3_characterizeImage_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8ab9f918-e3b3-451a-928b-2abd5fad9fe3_characterizeImage_1039768_61.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/48997116-3016-4ac2-8e20-9e1352e55b8d_isr_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/48997116-3016-4ac2-8e20-9e1352e55b8d_isr_1039768_61.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 217 try 0 failed parsl.dataflow.dflow INFO: Task 217 marked for retry parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash f5e836b0f93fee4fad4f087b7911a2d5 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 1 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:38:59 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613591ce2cda65bcdaa33f401dbf71bb with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 83b6e642059045b600582fea958eb446 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8419c5ca-655c-4f2e-9276-d1b509b7d5df_characterizeImage_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8419c5ca-655c-4f2e-9276-d1b509b7d5df_characterizeImage_1039768_49.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/5918e687-f605-4d77-a267-7369c56ee10a_isr_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/5918e687-f605-4d77-a267-7369c56ee10a_isr_1039768_49.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3734540d0bdc97e6cc8dac96d1d57a with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 59dde85e1ec413f367a2b6261a9a7019 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2789a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f2e26fe4-8570-42e3-87db-408deb5de67f_characterizeImage_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f2e26fe4-8570-42e3-87db-408deb5de67f_characterizeImage_1039769_61.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ff74e4d1-b2d3-47eb-8809-f35098480533_isr_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ff74e4d1-b2d3-47eb-8809-f35098480533_isr_1039769_61.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 985a2b4754d53596fa02e64c5f677e49 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash c4f0573ebf6b2feee9d649dfdc012e5d parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a15267d4-26e2-42aa-a1e8-4b7479915d97_characterizeImage_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a15267d4-26e2-42aa-a1e8-4b7479915d97_characterizeImage_1039768_51.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f82265ea-5ca6-4a4b-8dad-91a751073a44_isr_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f82265ea-5ca6-4a4b-8dad-91a751073a44_isr_1039768_51.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60b40379e7566bdaf5f693cff21cccb6 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 3b7e599a170b496970693f6bda3ba2ee parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6933caf3-f0c9-47b2-a35c-83c7b22e4a43_characterizeImage_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6933caf3-f0c9-47b2-a35c-83c7b22e4a43_characterizeImage_1039672_35.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/91cc2541-7795-446e-826c-63a5287a113a_isr_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/91cc2541-7795-446e-826c-63a5287a113a_isr_1039672_35.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 378557d1475d2cf9bf3dabfbe71f3849 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash c70556e45370727a098b69eccd25bb91 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4892641c-ae00-4f09-9493-826c8ae8f19b_characterizeImage_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4892641c-ae00-4f09-9493-826c8ae8f19b_characterizeImage_1039674_20.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/ca6aed33-352d-430a-a211-5c41e798fc4c_isr_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/ca6aed33-352d-430a-a211-5c41e798fc4c_isr_1039674_20.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f55d97f93320cf74dc6d4bf985dce39 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash c76faa5d3a0e67fbc1fac0233655d27f parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c42385a1-28e7-47e7-881b-f1db37647fde_characterizeImage_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c42385a1-28e7-47e7-881b-f1db37647fde_characterizeImage_1039674_28.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/644833b5-4463-474a-be28-0f38abf3b278_isr_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/644833b5-4463-474a-be28-0f38abf3b278_isr_1039674_28.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8c809bb5eb841212d2b49bbe922a21d with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash 0ea229f37cfd66cba6f21e4e4bef4724 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c60f5355-119c-43e1-b577-46e6a179336f_characterizeImage_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c60f5355-119c-43e1-b577-46e6a179336f_characterizeImage_1039674_60.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c9ea411a-5a65-47f9-a5ef-1323e6bc74d1_isr_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c9ea411a-5a65-47f9-a5ef-1323e6bc74d1_isr_1039674_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:00 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b1c6f5902ae364e603beb3e5fe102c3 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash 65e4896610348f173f05a01b9ddc33ab parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/04755abd-ba41-4f8c-97a9-72f1ba1bcd4f_characterizeImage_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/04755abd-ba41-4f8c-97a9-72f1ba1bcd4f_characterizeImage_1039673_19.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4d2a6fa7-3aaf-4b6b-a39b-5d2ba1546a0b_isr_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4d2a6fa7-3aaf-4b6b-a39b-5d2ba1546a0b_isr_1039673_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 557ae374a49047a430e30a52b7d7ee5c with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 52b4810f94b01ee8125cb1d81203898d parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c6cf1c7d-70b7-43cf-84d6-947b174f76ac_characterizeImage_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c6cf1c7d-70b7-43cf-84d6-947b174f76ac_characterizeImage_1039673_29.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/bf0dd6cb-a17c-4698-b36a-a6e9adfb5fea_isr_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/bf0dd6cb-a17c-4698-b36a-a6e9adfb5fea_isr_1039673_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:01 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae58bffd32a4050c6160baddf9db1d41 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 9f7b3c4e452b369b22039683eaa33f83 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/40f3aa90-20a5-4462-adc1-0285c2a9dbdf_characterizeImage_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/40f3aa90-20a5-4462-adc1-0285c2a9dbdf_characterizeImage_1039769_45.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3262cd85-d5c8-4d6e-8da1-6bbe17df7135_isr_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3262cd85-d5c8-4d6e-8da1-6bbe17df7135_isr_1039769_45.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08476dc303839bc162e90f616e85101 with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash ba7c34d13c812b47d789ea717e00c278 parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c3be8e51-6bb3-4f32-aa0c-9328cc6405bf_characterizeImage_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c3be8e51-6bb3-4f32-aa0c-9328cc6405bf_characterizeImage_1039769_35.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8b2cc1b3-8222-4d38-b961-9a81a89bb2c0_isr_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8b2cc1b3-8222-4d38-b961-9a81a89bb2c0_isr_1039769_35.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7801e3e109516c0d9e11d45b4009e2a5 with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash fdff5b380a29ad5a5d621d97eb8892bc parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cb980859-ba57-4d6b-b4c7-8b32f26031b0_characterizeImage_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cb980859-ba57-4d6b-b4c7-8b32f26031b0_characterizeImage_1039673_44.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/be4ecd6e-de8a-486f-9cd6-81cfd575c905_isr_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/be4ecd6e-de8a-486f-9cd6-81cfd575c905_isr_1039673_44.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fedb1fdc5858d743600861fc6e1ca8d with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 75be264c35a0bc598485ce02d0027513 parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/27006bec-3b06-4568-9a5b-bcf39a0827a1_characterizeImage_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/27006bec-3b06-4568-9a5b-bcf39a0827a1_characterizeImage_1039673_9.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d30de913-0188-4c08-bb68-11f86f6e744e_isr_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/d30de913-0188-4c08-bb68-11f86f6e744e_isr_1039673_9.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 207650b250abd469c6724f5411a158ff with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 0ac3b1af1a69d757e79dbc95eb85a594 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b5e0ad22-0273-4a03-b5b1-a8cd7bdb8dab_characterizeImage_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b5e0ad22-0273-4a03-b5b1-a8cd7bdb8dab_characterizeImage_1039769_25.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/15c1dce2-744b-4e38-ad58-bc6576e2fb9a_isr_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/15c1dce2-744b-4e38-ad58-bc6576e2fb9a_isr_1039769_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:02 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed96863a0dc57f4ea080d27138f3079 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 0caacf0273fdbbe570c4fa5ddba7ac7c parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9515a74a-9775-4b44-b445-a82d76d1b270_characterizeImage_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9515a74a-9775-4b44-b445-a82d76d1b270_characterizeImage_1039672_32.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c86d0612-1f4f-4b33-8b92-5f01c99f5bab_isr_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/c86d0612-1f4f-4b33-8b92-5f01c99f5bab_isr_1039672_32.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ba211e975d665a8486f1594a15efe7 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 1cd2db4faa7ac67b66ea86a62097fab2 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b22bad5b-96a6-4b50-bc58-22416922b735_characterizeImage_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b22bad5b-96a6-4b50-bc58-22416922b735_characterizeImage_1039673_60.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/73ec8257-1066-4bd1-b289-976376a216bd_isr_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/73ec8257-1066-4bd1-b289-976376a216bd_isr_1039673_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8c88f6b9c46ccb1f1035ae9e6aaa8a6 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 9d2309218952b47cc2d696797c19fa62 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c4129ac7-06c9-422c-84a4-1059f541ab3a_characterizeImage_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c4129ac7-06c9-422c-84a4-1059f541ab3a_characterizeImage_1039769_20.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9647c00e-fd45-4fd2-8366-ed86939456be_isr_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9647c00e-fd45-4fd2-8366-ed86939456be_isr_1039769_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee9960a2283647d9b7d740baf68c747 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 2754608e98c16b5b6b5b0c1a5b1035f1 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f75a726f-c6a1-4e75-98ce-4959fdc7ff41_characterizeImage_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f75a726f-c6a1-4e75-98ce-4959fdc7ff41_characterizeImage_1039672_10.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a9d41983-a25b-4b96-b0fe-6ff710399fde_isr_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a9d41983-a25b-4b96-b0fe-6ff710399fde_isr_1039672_10.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8277974bec75bbc6b82d0e32b4d94d29 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 8134c62cac0bbef1e740930c513727bd parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/20b74a1a-a52c-4de2-920e-6461d2da2ebd_characterizeImage_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/20b74a1a-a52c-4de2-920e-6461d2da2ebd_characterizeImage_1039674_27.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a31d5ced-9faf-46e6-b720-6e4110fa6555_isr_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a31d5ced-9faf-46e6-b720-6e4110fa6555_isr_1039674_27.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 827ba72fa64cf5f6de5f3f2d33d301f2 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash 85b1b0af0af2364ff369fe4441951d8d parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5936482b-62b0-4b05-b5f6-5f13a009938b_characterizeImage_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5936482b-62b0-4b05-b5f6-5f13a009938b_characterizeImage_1039768_38.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9b93d64a-eb02-49e6-a389-327b744ea61a_isr_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9b93d64a-eb02-49e6-a389-327b744ea61a_isr_1039768_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:03 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41738c39e170c2d546fed454d8fd0649 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash c8938c165f04a041ed31c3d701bb5811 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d3f2d61-5942-4cd7-bac4-a317ac2fea6b_characterizeImage_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d3f2d61-5942-4cd7-bac4-a317ac2fea6b_characterizeImage_1039769_8.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f050114f-9a20-4038-84d8-5d77ecefeb07_isr_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f050114f-9a20-4038-84d8-5d77ecefeb07_isr_1039769_8.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 591fb99b20a230b54b8708c9f8bda4b5 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 489a7161106e94afb16c1dfa80c84b6f parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7cd06872-25a9-4522-9efd-eb4334a893aa_characterizeImage_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7cd06872-25a9-4522-9efd-eb4334a893aa_characterizeImage_1039673_24.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9c0866af-5e47-4d16-bc21-f9eb9274b415_isr_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/9c0866af-5e47-4d16-bc21-f9eb9274b415_isr_1039673_24.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3bd777e3ee2bea868f38811ff5dc2e4 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 37a881a5152f1e6eb48db27247533c34 parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/dbba8623-79cc-45cd-87dc-c539037c9d2d_characterizeImage_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/dbba8623-79cc-45cd-87dc-c539037c9d2d_characterizeImage_1039674_8.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/557aa162-a6c3-4ab9-bf54-760744381707_isr_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/557aa162-a6c3-4ab9-bf54-760744381707_isr_1039674_8.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0848e3310bf32a29e36b5421138a1e9f with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 215fd42eafaf1aa3e3f763745dafd8d4 parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/15517832-c596-4197-9d0e-3ac701f31355_characterizeImage_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/15517832-c596-4197-9d0e-3ac701f31355_characterizeImage_1039674_9.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c8a8db4b-2e95-4717-aa56-8837b63ecc0f_isr_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c8a8db4b-2e95-4717-aa56-8837b63ecc0f_isr_1039674_9.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b556f3a26326629902af20a68262d3cb with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 27848a2d9a80dbb73bcc238e319efc16 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d5a3ec6f-a22b-466b-9be4-de88abc68992_characterizeImage_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d5a3ec6f-a22b-466b-9be4-de88abc68992_characterizeImage_1039770_52.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f00ce571-cd13-4753-957e-0e3d96f4da22_isr_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f00ce571-cd13-4753-957e-0e3d96f4da22_isr_1039770_52.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80baa260ffd1c5f356282ecdf5954007 with result from task 693 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:04 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 53a2f08d22945d0200d968fc3297a0cc parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a198abf0-275c-4962-8827-a2885e602d67_characterizeImage_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a198abf0-275c-4962-8827-a2885e602d67_characterizeImage_1039770_4.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/83d5ea87-6d5f-443f-a9c6-abb6b2080ace_isr_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/83d5ea87-6d5f-443f-a9c6-abb6b2080ace_isr_1039770_4.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d499e0447346e7609cda2563aa9d93df with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 5ce401e43a1970a812fdb048fbdd16df parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4d3795fa-de41-4aa3-b1db-1c0982a305cd_characterizeImage_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4d3795fa-de41-4aa3-b1db-1c0982a305cd_characterizeImage_1039674_49.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a9dc094-7d86-4b5d-a1b8-7e064f60020d_isr_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a9dc094-7d86-4b5d-a1b8-7e064f60020d_isr_1039674_49.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 610cca6e1a5421c5d31784bc8004df22 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 773e794af1ac46a51071594b5f4a1817 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9d835180-73fb-4e6c-b680-5070339b403a_characterizeImage_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9d835180-73fb-4e6c-b680-5070339b403a_characterizeImage_1039673_49.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/662902c8-5f0f-40bd-bbb2-a5c6411ebef8_isr_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/662902c8-5f0f-40bd-bbb2-a5c6411ebef8_isr_1039673_49.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c852acc7e61510e38eaa623e34e3203c with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 10764c51c1ebc1a9d0c76678dfe94b30 parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/46cf6e8b-04d3-4a36-9d20-020a4085406f_characterizeImage_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/46cf6e8b-04d3-4a36-9d20-020a4085406f_characterizeImage_1039769_7.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/74fdd317-4fa2-47f3-880b-4777ab9f2db6_isr_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/74fdd317-4fa2-47f3-880b-4777ab9f2db6_isr_1039769_7.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0eda5fc73f50ec36925f682f851538dd with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 1e4e029e3638bed17fcac30be46f1d1a parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92c8ad5e-194a-48ed-aaec-84f82dc2b253_characterizeImage_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92c8ad5e-194a-48ed-aaec-84f82dc2b253_characterizeImage_1039672_59.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59b4980-9e30-4beb-ab3e-3314b9ed8d07_isr_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59b4980-9e30-4beb-ab3e-3314b9ed8d07_isr_1039672_59.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 879b5a73a278ed6c62c9828943395a40 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash fe775f872fc9b62b988329ceec4f5cbc parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/57ae3480-020f-474a-a8d5-b08aecf83014_characterizeImage_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/57ae3480-020f-474a-a8d5-b08aecf83014_characterizeImage_1039768_60.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/787a2710-880e-4f40-a3f2-553da330ea9d_isr_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/787a2710-880e-4f40-a3f2-553da330ea9d_isr_1039768_60.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56e44b7876189fca73e76b64afb8a39b with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 5f01de1bb9667bc29c65aa0b81aa2d6f parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fdd26a6-f37b-4b4f-9da3-aee8cba3d6e2_characterizeImage_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fdd26a6-f37b-4b4f-9da3-aee8cba3d6e2_characterizeImage_1039672_7.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/dd063d0c-ca7e-4920-bee7-c88c5492843e_isr_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/dd063d0c-ca7e-4920-bee7-c88c5492843e_isr_1039672_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:05 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:06 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:06 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:07 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:07 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:08 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:08 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:09 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:09 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:10 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:10 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:11 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82d9ea4c2a9cd148acb0e1177d368972 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash f93f177710c3e373b70af8b4b4769380 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4a922778-e8e7-4eb8-a275-b1820356a3be_characterizeImage_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4a922778-e8e7-4eb8-a275-b1820356a3be_characterizeImage_1039674_13.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/3dc521d9-5091-4b8c-b943-050334525fa3_isr_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/3dc521d9-5091-4b8c-b943-050334525fa3_isr_1039674_13.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 960fbbe3d16cebfec0c04c49e29870e9 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 3b8f20d39be0b5359ee326c43218954c parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2e3a89d1-fc94-4df4-8928-3af679da3e59_characterizeImage_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2e3a89d1-fc94-4df4-8928-3af679da3e59_characterizeImage_1039674_52.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd3590f6-d425-4cd3-aec5-eb1f63cb1d49_isr_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/fd3590f6-d425-4cd3-aec5-eb1f63cb1d49_isr_1039674_52.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0e849b5701470917ddab90cd301f4bb with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 849e4da7b13933bda45c77873e25be68 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0214e764-064b-4482-a3bf-651280fd530a_characterizeImage_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0214e764-064b-4482-a3bf-651280fd530a_characterizeImage_1039674_5.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/10caed27-4d66-4826-bb02-51ea92095ae9_isr_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/10caed27-4d66-4826-bb02-51ea92095ae9_isr_1039674_5.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f88ab049c5655777090ca1dff942e7b with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 51856b382adc62bac861247eec60aacc parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/54ad5496-cf2b-451d-8940-c87aa5142c20_characterizeImage_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/54ad5496-cf2b-451d-8940-c87aa5142c20_characterizeImage_1039674_61.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/5c69f005-3b6d-4d36-89ba-7a38134fca42_isr_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/5c69f005-3b6d-4d36-89ba-7a38134fca42_isr_1039674_61.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c320bcb7dc7d151c4db904df6811f6d with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 046be6b64817c3d97c6bf6e4a6c909b6 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/640e52a7-9885-4302-891a-9ebeb0cad2a4_characterizeImage_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/640e52a7-9885-4302-891a-9ebeb0cad2a4_characterizeImage_1039770_12.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/25a0fcec-2512-4d67-9190-5ca47646c8d1_isr_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/25a0fcec-2512-4d67-9190-5ca47646c8d1_isr_1039770_12.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad5f5dd65b47d4e0c6a0de949eb151cf with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash b6620c2b3bcfa1985ad0a9061f692c57 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f6946435-35e3-4f3b-846a-6750f7009f75_characterizeImage_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f6946435-35e3-4f3b-846a-6750f7009f75_characterizeImage_1039672_12.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/42d0dd8b-bdca-44b7-9d98-50f0993d3657_isr_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/42d0dd8b-bdca-44b7-9d98-50f0993d3657_isr_1039672_12.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9205a2a39ccf26a17042dcd45d5e7073 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 30fccf15ba7c9c67e8b0d57f8bfa80ba parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/cb8af2e1-d841-4ff6-9a11-81c40c98c97a_characterizeImage_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/cb8af2e1-d841-4ff6-9a11-81c40c98c97a_characterizeImage_1039769_47.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea489ef2-40f4-43be-b146-385a6b00d5e6_isr_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea489ef2-40f4-43be-b146-385a6b00d5e6_isr_1039769_47.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d966e094093e3ae4ba20728d865bc3 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 82da744ae84424c2420986ea505aa702 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/30aea6b1-daea-498f-a198-66c20a56095e_characterizeImage_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/30aea6b1-daea-498f-a198-66c20a56095e_characterizeImage_1039673_10.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/8096b4b2-734b-40b0-81c9-456ae2eaa67a_isr_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/8096b4b2-734b-40b0-81c9-456ae2eaa67a_isr_1039673_10.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 0 failed parsl.dataflow.dflow INFO: Task 34 marked for retry parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash d1ffef71e3498f373260a37e9f28ddb0 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 1 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 0 failed parsl.dataflow.dflow INFO: Task 70 marked for retry parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 4808d5e5f8ae0182bed8918210634ad8 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 1 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f9f8a2408210df19f4f5bc892d87fdb with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash ddcd73052b40c6d1e46da5f6da9c9cd0 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e394d4ed-321a-4e22-aea1-21ced761e996_characterizeImage_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e394d4ed-321a-4e22-aea1-21ced761e996_characterizeImage_1039674_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1500fdcd-486c-4cb5-8919-529f58a18a47_isr_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1500fdcd-486c-4cb5-8919-529f58a18a47_isr_1039674_32.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5eb87fcb9b72df33ccdbef5fee699d with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash fc6a5cbd37dc5785ebdeed396ed29d31 parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6411996d-0584-42d1-b63c-fa76777072e6_characterizeImage_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6411996d-0584-42d1-b63c-fa76777072e6_characterizeImage_1039770_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/347d60bf-472a-48db-86e3-1f28fe943904_isr_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/347d60bf-472a-48db-86e3-1f28fe943904_isr_1039770_31.stderr parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eeaf9ef92bd1c3431cceca09d992ca3 with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash 9548f222c8d1099a402ebcf7cf6e27f8 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c49f471-6b63-4656-a7ac-1ed5f4728afe_characterizeImage_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c49f471-6b63-4656-a7ac-1ed5f4728afe_characterizeImage_1039673_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/689a65a4-e35d-4bbf-9b7b-62e27d4291d5_isr_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/689a65a4-e35d-4bbf-9b7b-62e27d4291d5_isr_1039673_3.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0746f0fca9a658385577ba683b24886d with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 63914a521ff87c9f032f2b0b3641f4ff parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/75628134-4981-4882-b3b8-2cbffea08018_characterizeImage_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/75628134-4981-4882-b3b8-2cbffea08018_characterizeImage_1039672_41.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bbc446fd-7c38-4e38-a95a-15526254bd26_isr_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bbc446fd-7c38-4e38-a95a-15526254bd26_isr_1039672_41.stderr parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d99629aa790f115d3fdb62c95069c337 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash 65caedcbd8dbf64d1a9cd38c6e25b820 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/331767c9-6761-40bd-85fe-c8f7999d593d_characterizeImage_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/331767c9-6761-40bd-85fe-c8f7999d593d_characterizeImage_1039770_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/d871645a-480b-49a9-a9ed-3ead40782998_isr_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/d871645a-480b-49a9-a9ed-3ead40782998_isr_1039770_51.stderr parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4708b32c7aa54f7b051e50b276413076 with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash 9bfd13d2f9934e32583f9265c267c721 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/628ac53d-89cc-4daf-8589-f16e368d2df0_characterizeImage_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/628ac53d-89cc-4daf-8589-f16e368d2df0_characterizeImage_1039674_44.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d836b79c-d920-4683-b1eb-96200f2412c5_isr_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/d836b79c-d920-4683-b1eb-96200f2412c5_isr_1039674_44.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1751a00a9d659edd43effc37cf879b9 with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash e2fcdb309c62804131bbbe3fdf7e41e0 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2b44e1fa-c87b-49a2-945d-3e8c2bb9cd00_characterizeImage_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2b44e1fa-c87b-49a2-945d-3e8c2bb9cd00_characterizeImage_1039673_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f439c2d8-927f-4a60-b403-350ce4a82bce_isr_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f439c2d8-927f-4a60-b403-350ce4a82bce_isr_1039673_1.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2656fd2de2b856ed8246056965164d8f with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash e544aa688808e4abe1e3bab86c947396 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2154e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e5f1c46a-1fb9-463a-b01e-00927938397e_characterizeImage_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e5f1c46a-1fb9-463a-b01e-00927938397e_characterizeImage_1039674_7.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/841195da-8925-4dca-b863-dcc9f700c8af_isr_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/841195da-8925-4dca-b863-dcc9f700c8af_isr_1039674_7.stderr parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c20dbcecc1f99104ba64236a239694d1 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 70f0a972d6caa90b5e68204bc38a2ff8 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b503ee85-f6b9-402e-96a5-f0ab5d025df5_characterizeImage_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b503ee85-f6b9-402e-96a5-f0ab5d025df5_characterizeImage_1039674_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cf152fab-3e7e-4915-930d-dffe3fee3635_isr_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cf152fab-3e7e-4915-930d-dffe3fee3635_isr_1039674_29.stderr parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7980ac62042d8904d293f08614ddb91 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 693ca2412fc0984f7507df6cf0a0ea15 parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bb9efa5-5f78-4cd9-b6af-e4117611e83f_characterizeImage_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bb9efa5-5f78-4cd9-b6af-e4117611e83f_characterizeImage_1039769_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5728a37-6cce-4c8e-a1a6-4f4cda981a66_isr_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5728a37-6cce-4c8e-a1a6-4f4cda981a66_isr_1039769_23.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 622d745d45fbd3ce576bf9003b847243 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash b6827d60bce38eddd6434a6c6f7fbdb1 parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/5bed9fac-b870-4ebb-8f85-d39d8978cc98_characterizeImage_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/5bed9fac-b870-4ebb-8f85-d39d8978cc98_characterizeImage_1039672_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/aa0545e6-e57b-488d-8b9e-6a288b1e8a7c_isr_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/aa0545e6-e57b-488d-8b9e-6a288b1e8a7c_isr_1039672_30.stderr 2024-03-18 04:39:21 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:39:21 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:22 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:23 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e777381ef978a698fae778afd56fff0 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 5315936f6ee6e677d03fe2b2c9992228 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a9a15e4-2175-4f7f-92d6-4654a2a03473_characterizeImage_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a9a15e4-2175-4f7f-92d6-4654a2a03473_characterizeImage_1039672_22.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b2e38a1d-267f-43d8-b6bc-bc599573eb85_isr_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b2e38a1d-267f-43d8-b6bc-bc599573eb85_isr_1039672_22.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9be987897fbe7e2e11b73812d7eb874f with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 03edac2a7c16ab46dcdb21712cea95af parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93dbc31a-fb7b-4e15-a8c3-a1ad05c89ef3_characterizeImage_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93dbc31a-fb7b-4e15-a8c3-a1ad05c89ef3_characterizeImage_1039674_55.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/40a25c72-392c-4a5e-acc6-54406aa1a053_isr_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/40a25c72-392c-4a5e-acc6-54406aa1a053_isr_1039674_55.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9bb90145675890312ee78c30166d87 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 7bdd73091609b0893bd061b35132808d parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bfc7a977-e5e2-4f20-8b56-26997df86415_characterizeImage_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bfc7a977-e5e2-4f20-8b56-26997df86415_characterizeImage_1039770_6.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/59156357-8474-400b-b0c0-345dd98b2a1c_isr_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/59156357-8474-400b-b0c0-345dd98b2a1c_isr_1039770_6.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 979d086ea9a036f0188e054c2f58201f with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash d8d4ff1e38d2b69b89e217357462b220 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ccc570-0da4-41d4-926a-e97ccdde5960_characterizeImage_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ccc570-0da4-41d4-926a-e97ccdde5960_characterizeImage_1039770_9.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a525def4-d556-4794-b147-d33714396db5_isr_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a525def4-d556-4794-b147-d33714396db5_isr_1039770_9.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ee3c3f023de38c32c10f8bf04243a5c with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 9a909a29edb7df99e206b514ba55444a parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bf0ae2b-4b5c-46f5-b026-2d6c3f1145d6_characterizeImage_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bf0ae2b-4b5c-46f5-b026-2d6c3f1145d6_characterizeImage_1039769_15.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8e14eabb-e403-4d7d-bdb8-30774b38d67c_isr_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/8e14eabb-e403-4d7d-bdb8-30774b38d67c_isr_1039769_15.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4be5b801cc579bbc78082d84d7c7883f with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash ea30bac02e06b3e90a37cb50bc961a2a parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a7af225-f5a9-4334-9722-3770b7ab9f22_characterizeImage_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a7af225-f5a9-4334-9722-3770b7ab9f22_characterizeImage_1039672_48.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a30528b8-2fee-4a19-82b6-2740e72e798d_isr_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a30528b8-2fee-4a19-82b6-2740e72e798d_isr_1039672_48.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3b04a413a4f3798f0b7832f929575e4 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash db972311f8c01d9d72dda1fabd6542d4 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/dc9af798-997a-48ad-86b5-1ea947ecad56_characterizeImage_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/dc9af798-997a-48ad-86b5-1ea947ecad56_characterizeImage_1039770_42.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/dc591d2e-78a8-4253-8f4e-491bbf456fa9_isr_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/dc591d2e-78a8-4253-8f4e-491bbf456fa9_isr_1039770_42.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2dc10bc65f5c8e76bb40cae8cfb0960 with result from task 288 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:23 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 7328388aa90c10e0844ba035777a641c parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e658698d-1157-4f0b-b16e-6b1bd6953e18_characterizeImage_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e658698d-1157-4f0b-b16e-6b1bd6953e18_characterizeImage_1039769_30.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/781dd59c-34fb-438e-956b-e7f674183580_isr_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/781dd59c-34fb-438e-956b-e7f674183580_isr_1039769_30.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 428cc2721e5a1966ce67814b5c193b31 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 2a0ab0be7c58141b393a9327f7aea558 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4c4d87c7-79ac-4d58-ba74-f2c0be05ea00_characterizeImage_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4c4d87c7-79ac-4d58-ba74-f2c0be05ea00_characterizeImage_1039769_53.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1ed6b4e0-72a6-4088-8c96-95afe4dfbdfb_isr_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1ed6b4e0-72a6-4088-8c96-95afe4dfbdfb_isr_1039769_53.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14e582b158f286ea168fe44ffd3e60a7 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 49ccb8747a64984e12388f5420b66c4c parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2e17e90a-f61f-4126-9d86-fce3f3a9395c_characterizeImage_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2e17e90a-f61f-4126-9d86-fce3f3a9395c_characterizeImage_1039770_47.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b16192ca-b849-463a-849c-a91caa94e163_isr_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b16192ca-b849-463a-849c-a91caa94e163_isr_1039770_47.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e65a92b0e488568832c23f3715ad8f4 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 33b58fa52f670ebfa81f76393e416e2d parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3823e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b92e76c0-807d-4bce-81e8-636a8585a8e8_characterizeImage_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b92e76c0-807d-4bce-81e8-636a8585a8e8_characterizeImage_1039770_59.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3ed58890-7333-4dbe-94e9-cee946119578_isr_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/3ed58890-7333-4dbe-94e9-cee946119578_isr_1039770_59.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6629a34e50472512312971bec9201312 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 56a2fc500599a5f8ec64ac43c584dea5 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c67850ad-5d62-45e9-a8c0-866b52333020_characterizeImage_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c67850ad-5d62-45e9-a8c0-866b52333020_characterizeImage_1039673_26.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5a0a7c0d-39b8-4024-a41d-5fc95384ba56_isr_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5a0a7c0d-39b8-4024-a41d-5fc95384ba56_isr_1039673_26.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45467c72cad86429b2b205daa75a5337 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 98aeefe02e253befd74663dbe6c52c8a parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/654fe64a-de3f-4fa8-bee5-ee86f2a05f2f_characterizeImage_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/654fe64a-de3f-4fa8-bee5-ee86f2a05f2f_characterizeImage_1039672_43.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/502eb62b-ab25-47df-815a-4e654a325b46_isr_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/502eb62b-ab25-47df-815a-4e654a325b46_isr_1039672_43.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdfcdbf5f35858173f26c6dc48e0e0d5 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash d24654ee64c110620e7dd960e7ffe67a parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d793723-5fa3-49ee-a850-8e87e15a6484_characterizeImage_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d793723-5fa3-49ee-a850-8e87e15a6484_characterizeImage_1039769_55.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/429c9a8d-ac72-4ace-af59-202b3e455bcf_isr_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/429c9a8d-ac72-4ace-af59-202b3e455bcf_isr_1039769_55.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f530e67128509b33a1e207072f20c799 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash c35af38b53bc15e027208170e032ab83 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/30b4eeeb-c111-4ac4-9a31-ced76b9c394b_characterizeImage_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/30b4eeeb-c111-4ac4-9a31-ced76b9c394b_characterizeImage_1039672_8.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/306f8efc-0b19-4e46-8b36-32cbf41dafd0_isr_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/306f8efc-0b19-4e46-8b36-32cbf41dafd0_isr_1039672_8.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 969f995b9b7d45fba9b1a119c2a2a537 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash beb19f745a68f2e9cdbf2bb2c168a6ce parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/49d5f573-facb-4505-aba1-bc5f4fdd8573_characterizeImage_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/49d5f573-facb-4505-aba1-bc5f4fdd8573_characterizeImage_1039674_19.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b5912956-151e-4033-bd7f-0813b1a5d8b6_isr_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/b5912956-151e-4033-bd7f-0813b1a5d8b6_isr_1039674_19.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11ce4164b3f75db1363d1785580343da with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 161806d71284d6cdee7897630efff126 parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1f90a765-9460-4db2-a6a8-a562c75bd406_characterizeImage_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1f90a765-9460-4db2-a6a8-a562c75bd406_characterizeImage_1039770_55.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fd02e8bf-bb34-438c-a8b8-79fdf09066db_isr_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fd02e8bf-bb34-438c-a8b8-79fdf09066db_isr_1039770_55.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4256c4bf5138299faeb8158448b1606d with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash 1c969f7ac56af3655107c306e4c1e422 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6cd0d9f4-6e19-4bf4-85c9-f59fa7e34ff7_characterizeImage_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6cd0d9f4-6e19-4bf4-85c9-f59fa7e34ff7_characterizeImage_1039673_42.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/219fc9e7-9286-4931-bc74-301956d13214_isr_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/219fc9e7-9286-4931-bc74-301956d13214_isr_1039673_42.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c764d7cbe2212e1512f61568bd368bd3 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 43d495fb41b926d6ac3a0d2f8a9d5851 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cb8c69fa-ebef-4cc5-9af0-2b762e320026_characterizeImage_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cb8c69fa-ebef-4cc5-9af0-2b762e320026_characterizeImage_1039770_15.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a80acb93-f353-4af1-84a4-0e5d551313b2_isr_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a80acb93-f353-4af1-84a4-0e5d551313b2_isr_1039770_15.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8322e5cbbd6805167109a85cd2d7c5be with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 1d636eb396963e2db26354049da3fc34 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da06536f-1ccd-4fef-aa4e-2422646f00d6_characterizeImage_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da06536f-1ccd-4fef-aa4e-2422646f00d6_characterizeImage_1039674_33.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a5af4ba7-d5fd-4e70-8619-2f410cfd965c_isr_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/a5af4ba7-d5fd-4e70-8619-2f410cfd965c_isr_1039674_33.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32eb0e3d41b8455b443811f8fe44652 with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 744db058e4ad2f6d4ca1e99c305bd84a parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1b1c58bb-1a59-4e9d-96d5-be4b4556ba3e_characterizeImage_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1b1c58bb-1a59-4e9d-96d5-be4b4556ba3e_characterizeImage_1039672_26.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd251fc1-6fca-4cd5-9725-32107528b358_isr_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cd251fc1-6fca-4cd5-9725-32107528b358_isr_1039672_26.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1000b3ab8003933c0a656d6c98350279 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 2fe6d598f7d34c66405c47f067fd36e7 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/decbcd64-e7c6-4152-aa79-9e6967caa648_characterizeImage_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/decbcd64-e7c6-4152-aa79-9e6967caa648_characterizeImage_1039674_25.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1bc8dfb5-0ed7-4cdc-9bab-7cb139a9d355_isr_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1bc8dfb5-0ed7-4cdc-9bab-7cb139a9d355_isr_1039674_25.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65159713246cdef5c82702b2f0bd014d with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 5a5d95d86b6885eb265ecbdf3d3320f3 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/49cb8a90-c9cf-4a2e-b223-0aa6a72221bb_characterizeImage_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/49cb8a90-c9cf-4a2e-b223-0aa6a72221bb_characterizeImage_1039770_28.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/60a40a12-7025-452c-9c26-46cf6fa39108_isr_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/60a40a12-7025-452c-9c26-46cf6fa39108_isr_1039770_28.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4ebc44a3f65d55595b05a3cfd2c2910 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 6dcb2cf0e93f13c18045b527cf95fd06 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c664ebdb-8626-4050-91a2-b6b0de715653_characterizeImage_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c664ebdb-8626-4050-91a2-b6b0de715653_characterizeImage_1039673_12.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ba9b878e-5bdb-44f7-b67d-2f3e1b2ddaa1_isr_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ba9b878e-5bdb-44f7-b67d-2f3e1b2ddaa1_isr_1039673_12.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3d21bf02ef897a73a0106ef883bf38d with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 1bd366601fea6fb07609c5a855bf7e7b parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b6e3c96d-1407-423e-aacc-d2ed0bada66d_characterizeImage_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b6e3c96d-1407-423e-aacc-d2ed0bada66d_characterizeImage_1039674_58.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c75cad51-f644-4d28-9d69-1187de0cc0d6_isr_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c75cad51-f644-4d28-9d69-1187de0cc0d6_isr_1039674_58.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a007d2bebcbbf794a103298122ed2adf with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash a5901348ef2a74e9d08aba6aafba3b1e parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/378768ce-924b-43a3-b6a4-6b833e559ebb_characterizeImage_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/378768ce-924b-43a3-b6a4-6b833e559ebb_characterizeImage_1039769_18.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7de3940d-c6bf-4e98-8cee-65a39e3c1eb1_isr_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7de3940d-c6bf-4e98-8cee-65a39e3c1eb1_isr_1039769_18.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4932a3532f5a1f84449181750eca2aba with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 9b740939bf83e6852401e19233a300cc parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3c20682e-ac46-414e-90b3-bb76ff6d5861_characterizeImage_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3c20682e-ac46-414e-90b3-bb76ff6d5861_characterizeImage_1039769_29.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/d7bb4fed-c769-4206-b60c-3c6f93fdb9a6_isr_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/d7bb4fed-c769-4206-b60c-3c6f93fdb9a6_isr_1039769_29.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87981780ae4a5eb6b34b308cbf4dc64f with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash fe693c55571fca439a549e390cd8113b parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0349da11-5784-4dae-b050-2675f0dfe144_characterizeImage_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0349da11-5784-4dae-b050-2675f0dfe144_characterizeImage_1039768_28.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b29e3e8a-5451-4046-a326-07e4d0716113_isr_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b29e3e8a-5451-4046-a326-07e4d0716113_isr_1039768_28.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3a0453ebd03908a93e06dcdd5e4303 with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 05d35316c02c6f7bb6ef1c18e1c16cdf parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9b088661-0f38-4631-9ede-92ab43da17bd_characterizeImage_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9b088661-0f38-4631-9ede-92ab43da17bd_characterizeImage_1039674_16.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a72fb2f-524e-4f91-8db3-ddf2419306e6_isr_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1a72fb2f-524e-4f91-8db3-ddf2419306e6_isr_1039674_16.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be1a71ff794ac80d47b8db3b9f772bd with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 546ee9715678a632be2cc79306176626 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e889cafc-0a6a-438e-ac45-43cb7f5775a3_characterizeImage_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e889cafc-0a6a-438e-ac45-43cb7f5775a3_characterizeImage_1039768_29.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b8663665-6a24-4cef-9d2a-c86373e9eade_isr_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b8663665-6a24-4cef-9d2a-c86373e9eade_isr_1039768_29.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bdcd1ba383b269531f1fd7085eec9ed with result from task 312 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:24 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 1e86f4bbb1b01a8c8a81c8c73d4efe1c parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bb00c70a-1686-4414-aa25-73629b248351_characterizeImage_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bb00c70a-1686-4414-aa25-73629b248351_characterizeImage_1039673_31.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/01c43af4-1438-4e2f-a278-4d5a4ea028e0_isr_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/01c43af4-1438-4e2f-a278-4d5a4ea028e0_isr_1039673_31.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae72f3efb212ec649426e00571cbe9f2 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash e9cfbe29bbc2444969fb013b422391c4 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cfd347f2-062e-499a-b91f-f8c7249bcb9b_characterizeImage_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cfd347f2-062e-499a-b91f-f8c7249bcb9b_characterizeImage_1039673_52.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/6aecd294-7b09-4110-aa08-430393361b09_isr_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/6aecd294-7b09-4110-aa08-430393361b09_isr_1039673_52.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf2d6d1408795e2c94afcc87f808b64e with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 68692e1ec07fee151d000551b0a55050 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54ab28f6-949e-4780-9796-36da3e11d1dc_characterizeImage_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54ab28f6-949e-4780-9796-36da3e11d1dc_characterizeImage_1039770_50.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b536da77-1afa-4655-aab0-3a141e4bc738_isr_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b536da77-1afa-4655-aab0-3a141e4bc738_isr_1039770_50.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9c2c3f66515ca357809e534b019f338 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 94c5757c6590cc83325dd2fc4096e4a5 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/043d8722-05d9-4ce0-a6af-db81236cf69b_characterizeImage_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/043d8722-05d9-4ce0-a6af-db81236cf69b_characterizeImage_1039673_58.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d52ec15-95fd-463f-b459-c96e53d47963_isr_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d52ec15-95fd-463f-b459-c96e53d47963_isr_1039673_58.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b42f330d31da24bff54c1525d1d25c with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 9300a515b3c2918645c46caf0e27b516 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a66e2a5-c4f7-4920-9efe-b0264f3e8735_characterizeImage_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a66e2a5-c4f7-4920-9efe-b0264f3e8735_characterizeImage_1039768_20.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c8360159-eed8-4dd8-aa14-0dc94b91cb91_isr_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c8360159-eed8-4dd8-aa14-0dc94b91cb91_isr_1039768_20.stderr parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afc8d0526afbea45f20b5e226e6ddf15 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash b38751682943fbd25eb561c8bfebdc99 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/270c34ea-32b5-4166-b9de-2a07547121a1_characterizeImage_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/270c34ea-32b5-4166-b9de-2a07547121a1_characterizeImage_1039673_16.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4dea901f-bd07-4612-9eba-defd265a6abe_isr_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/4dea901f-bd07-4612-9eba-defd265a6abe_isr_1039673_16.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34a2d07d1e3ef45dda15431e8d7cced4 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 122ae793551358dba8e21ee60f330ee0 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3825c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e3a9db21-6d65-4bd6-9693-8e4df45f034c_characterizeImage_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e3a9db21-6d65-4bd6-9693-8e4df45f034c_characterizeImage_1039768_6.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f809532-e050-45b1-965a-b3e78bef0e28_isr_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f809532-e050-45b1-965a-b3e78bef0e28_isr_1039768_6.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39e12091d986b8af2cd5494e4fdfaf0b with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash 471a6aba10cb913e9f4913061b48a067 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2832a6ed-fc93-45a5-8f4c-7b0f7ce8aa11_characterizeImage_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2832a6ed-fc93-45a5-8f4c-7b0f7ce8aa11_characterizeImage_1039770_60.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/048c9b7d-a012-420d-91b8-1fec876062f8_isr_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/048c9b7d-a012-420d-91b8-1fec876062f8_isr_1039770_60.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cec42fc0141a154de1c948fed2b269d with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 1c5987ebb543f7200a826f169fb817c8 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2ef04e56-5442-4573-bd79-18c41c0ad30e_characterizeImage_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2ef04e56-5442-4573-bd79-18c41c0ad30e_characterizeImage_1039673_34.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f5903255-b522-4741-b516-3d60e8a2a143_isr_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f5903255-b522-4741-b516-3d60e8a2a143_isr_1039673_34.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4620c470ecc10b614be59581d9d36d85 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 4e859b41810cb56f02cb1ed7d8ad45d3 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/2becc26d-59ad-454b-a1ba-80a1967b7cba_characterizeImage_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/2becc26d-59ad-454b-a1ba-80a1967b7cba_characterizeImage_1039769_44.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fb781d59-f291-40aa-a4b4-855b7ded74ec_isr_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fb781d59-f291-40aa-a4b4-855b7ded74ec_isr_1039769_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:24 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 711286ec9be1135b09fa5b8d94750894 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 98a1b0f44ae04ba96a6bfd8f73516ea1 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7bc1b523-797e-4e90-9360-a3cbf17e1098_characterizeImage_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7bc1b523-797e-4e90-9360-a3cbf17e1098_characterizeImage_1039672_55.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/db16e0c6-d593-4fe7-9f23-c09418d554dc_isr_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/db16e0c6-d593-4fe7-9f23-c09418d554dc_isr_1039672_55.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7403f35239a8019f57bcf6efa2b2764 with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash fb74265a235cebb32fae25c2dfc71ee9 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/62899b5c-ce46-4ab9-a028-76362d400814_characterizeImage_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/62899b5c-ce46-4ab9-a028-76362d400814_characterizeImage_1039769_56.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/91411994-0b8f-4c66-b1de-a3180a2510eb_isr_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/91411994-0b8f-4c66-b1de-a3180a2510eb_isr_1039769_56.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9517171ab9d85da1d3d819710ffc6564 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash b10ff0d0839d12ba7652921f17b7bba3 parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0222b0b6-d361-4995-8fbe-2a03a0bdad01_characterizeImage_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0222b0b6-d361-4995-8fbe-2a03a0bdad01_characterizeImage_1039769_52.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/12d65d0b-7450-428d-b879-0182b7100009_isr_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/12d65d0b-7450-428d-b879-0182b7100009_isr_1039769_52.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e876576eedf78f81970558d0c2f971d2 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 3822e4c6f19ff0c234e0f9f7d03a5ca8 parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/47b164c6-9fa8-4a82-b960-6ce3780abb17_characterizeImage_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/47b164c6-9fa8-4a82-b960-6ce3780abb17_characterizeImage_1039769_42.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dd9dac18-1ca9-4fd0-975b-13b1eb7d4584_isr_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dd9dac18-1ca9-4fd0-975b-13b1eb7d4584_isr_1039769_42.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f6e68ff110664d3336f1f97b7ebc28a with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 80d5136c3b6ba9cf63c7e2fa4b13e73a parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a609c444-d528-4554-821e-41db6efe59d7_characterizeImage_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a609c444-d528-4554-821e-41db6efe59d7_characterizeImage_1039673_37.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5d873564-854d-4d81-a64d-9b35baeb805a_isr_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/5d873564-854d-4d81-a64d-9b35baeb805a_isr_1039673_37.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41a758426acc50337c6d093585401887 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash e7016e35d11232a2a5200562a77ff2c3 parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a84f70e7-e193-4f07-a521-7ccd66c33e57_characterizeImage_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a84f70e7-e193-4f07-a521-7ccd66c33e57_characterizeImage_1039770_54.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ff8dc757-22ed-4af5-9be3-ffd27b4d5c4f_isr_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ff8dc757-22ed-4af5-9be3-ffd27b4d5c4f_isr_1039770_54.stderr parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d31733971106d3b2fe11061ccbec1bf with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 5a2c6ac1106f22f020c79bcb06948ead parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a0879060-dcd3-4095-b7f5-bf0f5a63fb7c_characterizeImage_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a0879060-dcd3-4095-b7f5-bf0f5a63fb7c_characterizeImage_1039768_23.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/49da0891-3fab-4b73-87b0-a9176a9d28d9_isr_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/49da0891-3fab-4b73-87b0-a9176a9d28d9_isr_1039768_23.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ef171f1116a39ddf49365a953686baa with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 5763f669bba6b9f06de02d336bf18d01 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ead6dfb7-27b5-424a-9cdc-709c28c49dd9_characterizeImage_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ead6dfb7-27b5-424a-9cdc-709c28c49dd9_characterizeImage_1039768_7.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/903319e1-0242-40bc-bee6-8611fc3ac66a_isr_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/903319e1-0242-40bc-bee6-8611fc3ac66a_isr_1039768_7.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a528fd2ca5161dc91b6a9c906c75fb4 with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 82ae5edb1b187df355264224a8718b10 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d6b4d24c-bd27-4337-b69f-e5dfc21b2ae4_characterizeImage_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d6b4d24c-bd27-4337-b69f-e5dfc21b2ae4_characterizeImage_1039770_38.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/71a322a9-869f-42cc-9e83-481299d8edc2_isr_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/71a322a9-869f-42cc-9e83-481299d8edc2_isr_1039770_38.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21682182283a765dfccc07656e20e357 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 3e20231e8c6a2dce3da37b256c270c79 parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/68b0036a-64b4-4fbd-aab0-1019e0d65edd_characterizeImage_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/68b0036a-64b4-4fbd-aab0-1019e0d65edd_characterizeImage_1039672_44.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/60badfaf-c676-4c41-9c42-26db9f3de269_isr_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/60badfaf-c676-4c41-9c42-26db9f3de269_isr_1039672_44.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0fafe07f6056a9be716dc59be067a8a with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 19e06ef2c59148a07fcab3375b440048 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5bec6aea-cb92-49c4-b9dc-cfdc6c375e8a_characterizeImage_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5bec6aea-cb92-49c4-b9dc-cfdc6c375e8a_characterizeImage_1039768_34.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/41fc38b5-9d33-4bd5-bd10-4f6d1fcf8dc4_isr_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/41fc38b5-9d33-4bd5-bd10-4f6d1fcf8dc4_isr_1039768_34.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ef110f161cdf677dff26675fcef50f5 with result from task 612 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:25 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 2935c9e679f24daeb71ca60698ef1fe9 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/7b5fb8e9-a9fb-4bbb-b86b-9521dac776f6_characterizeImage_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/7b5fb8e9-a9fb-4bbb-b86b-9521dac776f6_characterizeImage_1039768_30.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c076dacf-a42f-4627-a753-6fa4d62212e6_isr_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c076dacf-a42f-4627-a753-6fa4d62212e6_isr_1039768_30.stderr parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96178167632ca9d23b9a60bdffc760ff with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash d0c0856f1b1d68652471a0d176d07d37 parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a5aeb4e1-1a4c-4f43-871f-05d2b3809c51_characterizeImage_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a5aeb4e1-1a4c-4f43-871f-05d2b3809c51_characterizeImage_1039673_59.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a4463dee-a82d-406d-bb06-8606ec21b0cc_isr_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a4463dee-a82d-406d-bb06-8606ec21b0cc_isr_1039673_59.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a31014b19a790067c22d7576fb25605 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 574e6fb5e4bdf9b6bcd561e45156e7ec parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/00117804-6b22-4003-ac41-6070cc9375c6_characterizeImage_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/00117804-6b22-4003-ac41-6070cc9375c6_characterizeImage_1039768_36.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e4a4188c-8dc2-4782-8a60-ea86fc27c832_isr_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e4a4188c-8dc2-4782-8a60-ea86fc27c832_isr_1039768_36.stderr parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01b908caad899f07db3dfa26849dd8d9 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash 5fa57a7a23038a5ab6293134eec52dc5 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e2805d44-c061-4d2f-a76a-652ca5f9de27_characterizeImage_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e2805d44-c061-4d2f-a76a-652ca5f9de27_characterizeImage_1039768_17.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/906cb417-a74d-4475-85c9-270373ae78ba_isr_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/906cb417-a74d-4475-85c9-270373ae78ba_isr_1039768_17.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2449b0beac48912b3b94a934818fdfc7 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash 069dd271893719e691c7eff4fefc1ac3 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0ac3a32b-26bc-4e46-8a36-9ebc7a7c3ad1_characterizeImage_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0ac3a32b-26bc-4e46-8a36-9ebc7a7c3ad1_characterizeImage_1039674_34.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4471920d-a424-4590-97d9-3993bf080fbc_isr_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4471920d-a424-4590-97d9-3993bf080fbc_isr_1039674_34.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 556e1d432399448b76bc530318353a15 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash ad2529f37a0929510ad417c724bdf75e parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/3f5ce35b-1ae5-4f05-a2b5-8a7f8a67f0c5_characterizeImage_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/3f5ce35b-1ae5-4f05-a2b5-8a7f8a67f0c5_characterizeImage_1039768_47.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6923542c-ba54-4bec-a5b6-9749750c8c47_isr_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6923542c-ba54-4bec-a5b6-9749750c8c47_isr_1039768_47.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71268a912f102563ef5bc3da8e7a248d with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 5c883b7788613707bd37c85f6e7ed588 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d02b06b2-5d62-4440-a94d-43eb6eeefe6c_characterizeImage_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d02b06b2-5d62-4440-a94d-43eb6eeefe6c_characterizeImage_1039769_13.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/612e4672-4a17-4141-9e55-50ab9bc331d4_isr_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/612e4672-4a17-4141-9e55-50ab9bc331d4_isr_1039769_13.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc73c46042789e6e9aba527b27eaec86 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 9410a9d2b87b964525f047174a401e29 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4be71972-336f-4d64-8f47-e5b5b76178dd_characterizeImage_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4be71972-336f-4d64-8f47-e5b5b76178dd_characterizeImage_1039673_57.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/cb562569-fa7d-4593-bd77-334041b9e71f_isr_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/cb562569-fa7d-4593-bd77-334041b9e71f_isr_1039673_57.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf4d4afbf49e1fb21d28591657877736 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 35a41a47b152c46f23622286e99530cd parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9c499b82-717e-4b9d-b93c-981e452e0411_characterizeImage_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9c499b82-717e-4b9d-b93c-981e452e0411_characterizeImage_1039674_46.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f31173e9-98f6-40ee-95e1-b976c204f41b_isr_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f31173e9-98f6-40ee-95e1-b976c204f41b_isr_1039674_46.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e4aa2fa6df61927e61247bb04905350 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash d5f6b9a128747c74b8cc50aac6831aaf parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/3ab236f9-7f6d-45c6-b418-2df4fb754383_characterizeImage_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/3ab236f9-7f6d-45c6-b418-2df4fb754383_characterizeImage_1039674_54.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8551df42-f3eb-42a8-a95c-6ac09df51e80_isr_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8551df42-f3eb-42a8-a95c-6ac09df51e80_isr_1039674_54.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71416e5771461bfdb98d526c07525bf1 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash fe87af87c4ed36fde0f4c87d0fddd5c3 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3baa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/c12e208f-a8cc-4a4d-a4e8-03c14105dbe2_characterizeImage_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/c12e208f-a8cc-4a4d-a4e8-03c14105dbe2_characterizeImage_1039768_57.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1248a035-ddae-4abc-b020-f7f3d7eaedee_isr_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1248a035-ddae-4abc-b020-f7f3d7eaedee_isr_1039768_57.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e84a15520f58f3e28a3afbf0439e9959 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash fdba461ba7e1d803857f48539bfe4dc9 parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/7291f0b1-5734-4d5b-848c-3d52fcc3370d_characterizeImage_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/7291f0b1-5734-4d5b-848c-3d52fcc3370d_characterizeImage_1039674_17.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/34e8d805-5a4c-4263-8c50-6cd7f6a500a0_isr_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/34e8d805-5a4c-4263-8c50-6cd7f6a500a0_isr_1039674_17.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a04e03a8871cebb92f9a0727259dbc8 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash b8bb46357387d987630f29d8a7c9055d parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4a7572b7-6233-4820-b98f-8792a8aeecb7_characterizeImage_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4a7572b7-6233-4820-b98f-8792a8aeecb7_characterizeImage_1039673_53.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3c8b161-0ea4-4604-a28b-5b3a322c0365_isr_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3c8b161-0ea4-4604-a28b-5b3a322c0365_isr_1039673_53.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e256f22aad5938852cf7d08596cdd8f with result from task 531 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:25 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 43108c18e8f2767d1cfadfbd81f6871a parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/816d6fa7-ed55-45f7-ba9f-0526f409dbff_characterizeImage_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/816d6fa7-ed55-45f7-ba9f-0526f409dbff_characterizeImage_1039769_1.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5a61698-960c-48cf-a0b1-cc85f88e5d48_isr_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f5a61698-960c-48cf-a0b1-cc85f88e5d48_isr_1039769_1.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81d3d6ea450697426bcdcdcd5afb4db9 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 1fdddaa6698d7e9cdf0c93f4edc11f58 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/aa32f058-1b8b-4d76-b4bc-5e11d1283a51_characterizeImage_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/aa32f058-1b8b-4d76-b4bc-5e11d1283a51_characterizeImage_1039769_5.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b4cd7daa-f1bd-4737-94de-c63546f9c1eb_isr_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/b4cd7daa-f1bd-4737-94de-c63546f9c1eb_isr_1039769_5.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e8f3f7736c0c1bb1692d9338a92a230 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash e632633f14463af31324a34d0a4a3477 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/89bd3e8f-8c7c-4243-aa06-4ec85d86807e_characterizeImage_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/89bd3e8f-8c7c-4243-aa06-4ec85d86807e_characterizeImage_1039674_43.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/62a05f4e-8dc3-41f9-997f-bec15eb5b3f1_isr_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/62a05f4e-8dc3-41f9-997f-bec15eb5b3f1_isr_1039674_43.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dca1802c690859a37fac06e009fe4e20 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 8a3a6937bbcd0966dc1590b9c77cb28e parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/da4f4f2a-9695-4500-b696-34239efd0398_characterizeImage_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/da4f4f2a-9695-4500-b696-34239efd0398_characterizeImage_1039769_54.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/a2fe2589-5603-42f8-8df4-5497ed3b4c19_isr_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/a2fe2589-5603-42f8-8df4-5497ed3b4c19_isr_1039769_54.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1a350fdf826a56f245b7ed325745233 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash d80a5dfe76249cc227aaa09c62255ad2 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ebbd6cd3-b82b-4d6c-a83a-a43c370a9ef9_characterizeImage_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ebbd6cd3-b82b-4d6c-a83a-a43c370a9ef9_characterizeImage_1039770_25.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/231150fe-ea6b-43ab-8d72-127dffbf0e9b_isr_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/231150fe-ea6b-43ab-8d72-127dffbf0e9b_isr_1039770_25.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07b3e914d1fa706c3c40ab6c758375bf with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 344a97c984eb590c66ebd1bb79b49c2a parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b9d7197f-8e12-4286-b3c1-c08af633edaa_characterizeImage_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b9d7197f-8e12-4286-b3c1-c08af633edaa_characterizeImage_1039768_5.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/29eca76e-96ca-447d-bd25-5326933b5813_isr_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/29eca76e-96ca-447d-bd25-5326933b5813_isr_1039768_5.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 007a700f95057b1a627833bfa1e87d09 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 764d08e5b5d3cfa2eb14d397563f9b61 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f5f7e280-7b1e-454a-9921-b0d7060f511c_characterizeImage_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f5f7e280-7b1e-454a-9921-b0d7060f511c_characterizeImage_1039674_53.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/47808565-ca17-4ac5-b5c5-fec836c0ddcc_isr_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/47808565-ca17-4ac5-b5c5-fec836c0ddcc_isr_1039674_53.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d06653b0daed0a8cc6037fd7dedd1a3 with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash ed91303e2ccdfb86480b702962ffe410 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1c547a-9edf-41ab-be93-a0a63400248b_characterizeImage_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1c547a-9edf-41ab-be93-a0a63400248b_characterizeImage_1039673_15.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/472eb18f-23b8-461e-8c92-a059c59d2aed_isr_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/472eb18f-23b8-461e-8c92-a059c59d2aed_isr_1039673_15.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 737236f3de7916b63e3750a46062d508 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 9078dab436c51d405f2f42e588ce293f parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/17348e82-88bb-4c91-bc41-23e6d2573d5f_characterizeImage_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/17348e82-88bb-4c91-bc41-23e6d2573d5f_characterizeImage_1039672_38.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/601d4732-4bb1-4ea5-981e-eac5b404c853_isr_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/601d4732-4bb1-4ea5-981e-eac5b404c853_isr_1039672_38.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2bc47481a7de38121a65b00fd752c7cd with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash f4543b86129efbb377ad2762dcce31dd parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/96bec2ad-eb1b-4212-9345-6dcd4a4b3220_characterizeImage_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/96bec2ad-eb1b-4212-9345-6dcd4a4b3220_characterizeImage_1039674_14.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f4b8c501-5061-4df3-a0b5-3b1238e24da1_isr_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f4b8c501-5061-4df3-a0b5-3b1238e24da1_isr_1039674_14.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96b7e937a9078f24ff72f8ebe491904f with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash dc3f06337dbba2e36c095587afb8e9f6 parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9d2f316b-14a5-42ee-a45a-4bd101a6b5b7_characterizeImage_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9d2f316b-14a5-42ee-a45a-4bd101a6b5b7_characterizeImage_1039672_6.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f6ab3a65-6210-4b48-900e-5ee08eec9380_isr_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f6ab3a65-6210-4b48-900e-5ee08eec9380_isr_1039672_6.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d43c9a65475c4638822bb27c44faac with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 65b11b1bb58828c877b4b6d64d2aad13 parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9f2c1ef7-45a7-45c7-ab14-5a253453256a_characterizeImage_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9f2c1ef7-45a7-45c7-ab14-5a253453256a_characterizeImage_1039770_19.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f2d90025-b6b1-43ad-a74d-fbcad14d486a_isr_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/f2d90025-b6b1-43ad-a74d-fbcad14d486a_isr_1039770_19.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 146e8d2a15bdcc5ce24e21344cde39e0 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 6b0d48ae61e22737ffc3bb66bb9812ac parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/865fd6d3-3e50-4c5d-ae7e-5bd20d8478f2_characterizeImage_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/865fd6d3-3e50-4c5d-ae7e-5bd20d8478f2_characterizeImage_1039769_37.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0c6542bb-be87-438e-b4bd-645b5f2da134_isr_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0c6542bb-be87-438e-b4bd-645b5f2da134_isr_1039769_37.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1bdff674ef0c1e561936f69ba83a29 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash c340009f4938356cf90437d9ed672798 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ec4b12fc-4602-4daf-aa39-628192b0a4e5_characterizeImage_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ec4b12fc-4602-4daf-aa39-628192b0a4e5_characterizeImage_1039770_53.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e980629e-5268-4105-ace6-646b80d5a53e_isr_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e980629e-5268-4105-ace6-646b80d5a53e_isr_1039770_53.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d85ec86e9453135e42df42ec9329daf with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 0d5c48157d93f147eaddc2c0aecdccb6 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e6c629d9-750a-43da-a30b-cd33e4b1db96_characterizeImage_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e6c629d9-750a-43da-a30b-cd33e4b1db96_characterizeImage_1039768_10.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8df9be2e-536d-4217-838f-ae5e873af4c7_isr_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/8df9be2e-536d-4217-838f-ae5e873af4c7_isr_1039768_10.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d96db9f32baa72e9411f2f6ce047c1 with result from task 807 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:26 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 9b20e914281c9c562d25fa5cb9fc3655 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a2dd0dd9-ef9b-4590-a907-a97ff58a9abc_characterizeImage_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a2dd0dd9-ef9b-4590-a907-a97ff58a9abc_characterizeImage_1039770_23.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a4335ddc-a76a-421c-8c14-14556231ac9a_isr_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a4335ddc-a76a-421c-8c14-14556231ac9a_isr_1039770_23.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcade7b8c313cca389691f0ef538d808 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 3a89f1efb87f04f46684dfc6d961b03a parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/33f37876-183e-4734-a2c3-8f8d9440ca93_characterizeImage_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/33f37876-183e-4734-a2c3-8f8d9440ca93_characterizeImage_1039769_11.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea3a277e-fdd1-412d-b96d-7dfa54f88bf4_isr_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/ea3a277e-fdd1-412d-b96d-7dfa54f88bf4_isr_1039769_11.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 886bcb5a6a8f92fea16094ccf1952d97 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash 3be832005c17facf05d8395aff88e451 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/adcb998c-d55d-41a9-98a8-a47f698e39a9_characterizeImage_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/adcb998c-d55d-41a9-98a8-a47f698e39a9_characterizeImage_1039672_3.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/036e99ae-3f21-40ab-9b8e-0ba06a6f68b5_isr_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/036e99ae-3f21-40ab-9b8e-0ba06a6f68b5_isr_1039672_3.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 134c6ffc6e5da7aabdda54ebd840530d with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash e1045723deb8227d2e07a016fa972803 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/293ce291-69ec-46ea-8bf3-c9e006097a50_characterizeImage_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/293ce291-69ec-46ea-8bf3-c9e006097a50_characterizeImage_1039672_4.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/724ff8e4-4fa1-4a04-af3a-95a64e10ddc8_isr_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/724ff8e4-4fa1-4a04-af3a-95a64e10ddc8_isr_1039672_4.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bc7d4d79ae255cf1329a88135a42cef with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash bf25929da1c7849943c33f5ea2d235a3 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cceaf9b8-16f8-4fd0-beb7-5c3e6c70891d_characterizeImage_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cceaf9b8-16f8-4fd0-beb7-5c3e6c70891d_characterizeImage_1039770_7.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/384faf82-edce-4670-bcf0-b251f94dfa80_isr_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/384faf82-edce-4670-bcf0-b251f94dfa80_isr_1039770_7.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e3766b826501ea13b685484bc2f2d26 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 8a16d2bd3c766c3871bcc178d139adfd parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/efa506cc-28b4-457f-b7b9-b1e7b6b473bc_characterizeImage_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/efa506cc-28b4-457f-b7b9-b1e7b6b473bc_characterizeImage_1039768_22.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04cb2654-cddb-472a-b336-c7668054c4ac_isr_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/04cb2654-cddb-472a-b336-c7668054c4ac_isr_1039768_22.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 093fb7f7a0a3e69b010b1330141bb932 with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash aa31c1a370c9b277e4035560c83e4b9a parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/135a5858-6684-46bc-87b5-ba3bdd7dd7a4_characterizeImage_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/135a5858-6684-46bc-87b5-ba3bdd7dd7a4_characterizeImage_1039673_41.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c25b312f-ad2a-4657-b255-a52ac02bf333_isr_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/c25b312f-ad2a-4657-b255-a52ac02bf333_isr_1039673_41.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf034af50e803d421417b265ff66b5bb with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 4cb198a7265f5652cedc42fa82faaaff parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/69020a90-21ed-451e-a1ba-c987b0be982b_characterizeImage_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/69020a90-21ed-451e-a1ba-c987b0be982b_characterizeImage_1039672_24.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/39992be6-20d5-4b30-8d1f-ec24f6810c4e_isr_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/39992be6-20d5-4b30-8d1f-ec24f6810c4e_isr_1039672_24.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a0c3d17a312c1c0200b34642042531f with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 0cf3dc8eb0d663ff3ca36b1239b0edcc parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/63f2c2c7-f887-4ca2-8bd4-21ca10dec3af_characterizeImage_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/63f2c2c7-f887-4ca2-8bd4-21ca10dec3af_characterizeImage_1039768_45.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6123dc43-fc8f-419a-b486-14f1f2089100_isr_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/6123dc43-fc8f-419a-b486-14f1f2089100_isr_1039768_45.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad847e9ab7e5a6ecb0817852218808a5 with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 1276b3f9bd54415a04a8c80fef95c330 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce3fae6a-f1b0-493c-ba4f-992aebbc7a9e_characterizeImage_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce3fae6a-f1b0-493c-ba4f-992aebbc7a9e_characterizeImage_1039770_62.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/688f7553-639e-4017-879a-eb7f47740ceb_isr_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/688f7553-639e-4017-879a-eb7f47740ceb_isr_1039770_62.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15d3c55ea63a236bb030b77e78e99685 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash 60e1c938fed79ae7130ac548c94598b6 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a2230e5b-c1c2-4e01-9c54-b122072027b1_characterizeImage_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a2230e5b-c1c2-4e01-9c54-b122072027b1_characterizeImage_1039672_45.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f4d063b4-1500-4602-ab84-b0f2cb626d6e_isr_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/f4d063b4-1500-4602-ab84-b0f2cb626d6e_isr_1039672_45.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f8a8d866da14b45723b21e36837fe3 with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash fc8ad9df023ef62d1f2bb47857954ad7 parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d997cd36-1385-47a5-8047-3579e3cbbe78_characterizeImage_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d997cd36-1385-47a5-8047-3579e3cbbe78_characterizeImage_1039768_27.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/06ecab94-1f54-4b0e-a31c-fe6648551d06_isr_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/06ecab94-1f54-4b0e-a31c-fe6648551d06_isr_1039768_27.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b611e08efe538dc3cf59e0831a1612cc with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash a76ca5159e313af2abced1db52a3ed64 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/bcdee720-2ff7-4945-ab70-7b4a811e2a43_characterizeImage_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/bcdee720-2ff7-4945-ab70-7b4a811e2a43_characterizeImage_1039674_15.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/bf073c24-88be-4de3-ae73-53c83cb4068a_isr_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/bf073c24-88be-4de3-ae73-53c83cb4068a_isr_1039674_15.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53c9ba87714061da8487c2f1182c2468 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash f045fc907fe0afd68e1aeb2045f8dad3 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/064c05fc-3755-4b11-91e3-9741e17db24b_characterizeImage_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/064c05fc-3755-4b11-91e3-9741e17db24b_characterizeImage_1039672_47.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/938f8dd6-1c88-43dd-97be-6e052a4b06d3_isr_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/938f8dd6-1c88-43dd-97be-6e052a4b06d3_isr_1039672_47.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dadc3d5b798b9af95d2944912a081487 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash c51504a8f8898e5c818551cfabb0961d parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4171746f-4508-4ff4-8c25-f039d0cea85e_characterizeImage_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4171746f-4508-4ff4-8c25-f039d0cea85e_characterizeImage_1039770_5.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/cf4602d7-3634-4968-a662-aa7cd6b6b273_isr_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/cf4602d7-3634-4968-a662-aa7cd6b6b273_isr_1039770_5.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8c9e63374740e79b9793abd3b0399c0 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 057a92a02ed88565aa7f73a621e26e9b parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ceed348-2929-42a2-817e-2b6dab21df23_characterizeImage_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ceed348-2929-42a2-817e-2b6dab21df23_characterizeImage_1039672_14.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bef8a6bc-b24e-421b-a03c-071e29c63cdf_isr_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/bef8a6bc-b24e-421b-a03c-071e29c63cdf_isr_1039672_14.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 450c9b1eedc94456e8511591301037b3 with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 0789a807e5f644a3e15b125fdeb6452a parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ffe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdae1d56-5462-437d-81ce-47b3b2d63ba1_characterizeImage_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdae1d56-5462-437d-81ce-47b3b2d63ba1_characterizeImage_1039768_11.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c516a8cc-69b1-4bfc-b4d7-c6fb2e42bdc0_isr_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/c516a8cc-69b1-4bfc-b4d7-c6fb2e42bdc0_isr_1039768_11.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 771bdf85ea6ccd7f0198c8a3598c0692 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash c690d415efc70f1906353905eaf699f8 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f53ea1d1-371e-4944-862e-f3a33ca6c44b_characterizeImage_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f53ea1d1-371e-4944-862e-f3a33ca6c44b_characterizeImage_1039769_40.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/05587c87-52c6-4a31-bbc5-05eb3e0d5496_isr_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/05587c87-52c6-4a31-bbc5-05eb3e0d5496_isr_1039769_40.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1601450f91ea57261544a1377fec6115 with result from task 816 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:26 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 6ac77416945013922440ff9ce7e9351e parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9ea67fc6-6984-4963-a113-a5e9c5009ff4_characterizeImage_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9ea67fc6-6984-4963-a113-a5e9c5009ff4_characterizeImage_1039672_37.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d95f27c3-7a5c-4efb-b245-f6947358d59f_isr_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d95f27c3-7a5c-4efb-b245-f6947358d59f_isr_1039672_37.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17618b1c8c75901b32520ae81d3678f0 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 427c781ce231ab5c6fa0067582cdd0f6 parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/48ebba0d-27d3-4734-8480-f8d3733b90f1_characterizeImage_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/48ebba0d-27d3-4734-8480-f8d3733b90f1_characterizeImage_1039768_52.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/3be42370-0249-454e-aef1-d45c548ccb38_isr_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/3be42370-0249-454e-aef1-d45c548ccb38_isr_1039768_52.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f76526ecbe63f189e13f188930a7f4f with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash 254a27e4572ecb51515228287951c2b1 parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ae14ec0f-4d1b-4b4d-9b07-b8742109ee42_characterizeImage_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ae14ec0f-4d1b-4b4d-9b07-b8742109ee42_characterizeImage_1039768_35.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f0c8e658-f3f2-4e2e-92f6-e08af0665410_isr_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/f0c8e658-f3f2-4e2e-92f6-e08af0665410_isr_1039768_35.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efb6969ffa1696c0c2680e35e91dd404 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 9b027d4ad1022d0de89dda95f4b9625c parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ffb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/24c8414e-8979-4da1-8efb-528def55e7ec_characterizeImage_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/24c8414e-8979-4da1-8efb-528def55e7ec_characterizeImage_1039672_60.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/9aebd972-58b1-4cf4-a56a-733c4155b602_isr_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/9aebd972-58b1-4cf4-a56a-733c4155b602_isr_1039672_60.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d05906c6a118f061bee3ad7b391340fb with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash cabdc3ab43ca026e44624e543a98be0b parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4591304f-8a48-4316-9102-54f2bb23de73_characterizeImage_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4591304f-8a48-4316-9102-54f2bb23de73_characterizeImage_1039768_54.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/978dd359-a212-4098-a8bc-12ceba8320bf_isr_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/978dd359-a212-4098-a8bc-12ceba8320bf_isr_1039768_54.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d169db04a18742b75de34ea47199b9e0 with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 7799623fcd581f14738a27bd318423f3 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/30479a99-a593-462c-a06a-6dc2091f847d_isr_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/30479a99-a593-462c-a06a-6dc2091f847d_isr_1039674_2.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53364e04e8b1c04e38b25db831b4f38c with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash e5c73fe8eba524d833dec4ebd3731a00 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/731773ee-ab9d-4225-b462-a2b205435bc1_characterizeImage_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/731773ee-ab9d-4225-b462-a2b205435bc1_characterizeImage_1039770_44.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b61fbab0-8bd1-4e15-9ba4-708efd59fcdf_isr_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b61fbab0-8bd1-4e15-9ba4-708efd59fcdf_isr_1039770_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:26 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:27 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:27 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:28 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:29 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:29 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:31 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:31 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:32 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa45152d38d5bfae48ecb0511d99dc9 with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash a837c31cb6b4c0860e89de66bb85396d parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/98d0857a-07a6-40c1-9236-d239e3da8cd6_characterizeImage_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/98d0857a-07a6-40c1-9236-d239e3da8cd6_characterizeImage_1039769_9.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5e3d7eb3-7007-4ade-9d25-1ba6b42d4e32_isr_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/5e3d7eb3-7007-4ade-9d25-1ba6b42d4e32_isr_1039769_9.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de991897b55f6772688a9431eb0a99b6 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 0e65c8a1b3074993038fa8f73e36603e parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/31263d4b-4d2e-4aef-9994-866cf17ce023_characterizeImage_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/31263d4b-4d2e-4aef-9994-866cf17ce023_characterizeImage_1039770_27.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/4e8578f0-3cea-4d7d-bd31-c1a9ab22b921_isr_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/4e8578f0-3cea-4d7d-bd31-c1a9ab22b921_isr_1039770_27.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a94baa420ceaa961f9ece52b5dee698d with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 924a5786438e6e82b34c312a5ea094db parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/aaf10ef4-e8ca-47d1-a0ec-fd583db8a7f8_characterizeImage_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/aaf10ef4-e8ca-47d1-a0ec-fd583db8a7f8_characterizeImage_1039770_49.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ada42bfd-7ab4-46c2-8815-52eb567e8f16_isr_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ada42bfd-7ab4-46c2-8815-52eb567e8f16_isr_1039770_49.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45f4a40a3a3cd15b984b7c5855387333 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash a5d77854817e83d76b8c7abbc6d22cd2 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/73938340-9778-42a6-98d6-97646ebce3c2_characterizeImage_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/73938340-9778-42a6-98d6-97646ebce3c2_characterizeImage_1039673_54.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ce6ee819-0dbf-4d2c-8200-b1aa8176039e_isr_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ce6ee819-0dbf-4d2c-8200-b1aa8176039e_isr_1039673_54.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69a26dc2fe24f705892fe29a38c19afe with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 749395820f460226df75e32a439973aa parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/216c9984-6a11-4d93-991c-abf774f8861c_characterizeImage_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/216c9984-6a11-4d93-991c-abf774f8861c_characterizeImage_1039769_16.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f95d761c-ff47-40c2-8c1a-ab4a97cf8ed6_isr_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f95d761c-ff47-40c2-8c1a-ab4a97cf8ed6_isr_1039769_16.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4cb61fe00957a1e99aa0bef83c78fc6 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash ec58af8d6063cec4f9ef5276679b23e8 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bfbce4d5-c145-4810-907d-61a9971ecbe3_characterizeImage_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bfbce4d5-c145-4810-907d-61a9971ecbe3_characterizeImage_1039768_4.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f43f747-b97e-4b4d-9249-3fb0cef312b6_isr_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/2f43f747-b97e-4b4d-9249-3fb0cef312b6_isr_1039768_4.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d407ea5dde1e5d9c3a05ea768b9f8ec0 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash d03f7f1fed48a5ef6fa2b786ac4ba72c parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4cf765bf-6964-4cfb-b18d-a9fd9003350f_characterizeImage_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4cf765bf-6964-4cfb-b18d-a9fd9003350f_characterizeImage_1039769_19.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/19bdcc4f-2c1d-4760-9172-b8e3d3c4824c_isr_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/19bdcc4f-2c1d-4760-9172-b8e3d3c4824c_isr_1039769_19.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f261bd186147fe57734dc7e8170f7aab with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 6dd0a510695e662afb1906ac957a0e72 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/055541bd-aa93-4e36-b7f8-90fba62dd471_characterizeImage_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/055541bd-aa93-4e36-b7f8-90fba62dd471_characterizeImage_1039769_49.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c770f2d5-d935-44fa-b15f-5d495c9a5d1a_isr_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c770f2d5-d935-44fa-b15f-5d495c9a5d1a_isr_1039769_49.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a584e53cd0fc2228265d420478c4cd with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash cd50ad8ba0951b4d04e63fb6d1292c03 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/98ac3ef6-d8f1-43ab-89e9-5f5e314baf07_characterizeImage_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/98ac3ef6-d8f1-43ab-89e9-5f5e314baf07_characterizeImage_1039672_20.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59928f6-cf80-4041-b15a-e5d99c5ca2e7_isr_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/d59928f6-cf80-4041-b15a-e5d99c5ca2e7_isr_1039672_20.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6a3c4e91b36b4eae10001a84117977 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 87c02de5b7c4ca66568e1ca9167bfc0c parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4dc2af93-dc20-416a-a769-c8d891b1eb4f_characterizeImage_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4dc2af93-dc20-416a-a769-c8d891b1eb4f_characterizeImage_1039769_58.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fbe04133-9695-4edd-9286-2fc66d8695fb_isr_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/fbe04133-9695-4edd-9286-2fc66d8695fb_isr_1039769_58.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca18f3e8177e73bcc0ac5cbea64106e2 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash adae69e7475cfddbafa7207e09bf92d1 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/32cbb219-26fe-4b29-9ad6-f4f1dd8017d4_characterizeImage_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/32cbb219-26fe-4b29-9ad6-f4f1dd8017d4_characterizeImage_1039673_38.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/dce2b500-599b-4ffc-a35b-2cd415642d33_isr_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/dce2b500-599b-4ffc-a35b-2cd415642d33_isr_1039673_38.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15105021b8a7379292ac178af5bb6d31 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash e776f7b87610cae8a25b36d64609cff0 parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/63fe4c09-0eae-4edf-a839-284b8d2407a4_characterizeImage_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/63fe4c09-0eae-4edf-a839-284b8d2407a4_characterizeImage_1039769_6.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7f25bd96-3e09-4cc3-a7fd-f21651e09f0e_isr_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7f25bd96-3e09-4cc3-a7fd-f21651e09f0e_isr_1039769_6.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5579ab8cb1510a3b82ec472302c47e13 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 94f6e2973b880174a325d4619016483f parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d7a51b0b-deae-4721-ae8d-30422f79a75e_characterizeImage_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d7a51b0b-deae-4721-ae8d-30422f79a75e_characterizeImage_1039769_22.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9b845887-c59d-4320-9d03-7af7e82b442c_isr_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/9b845887-c59d-4320-9d03-7af7e82b442c_isr_1039769_22.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd612636db242caefd595364051a219c with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 0434ae480211595d5fa2a12c13a21f93 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ca4b8df1-b095-4459-afd8-51afd15c723f_isr_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/ca4b8df1-b095-4459-afd8-51afd15c723f_isr_1039770_2.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceccc9735db3246d7b5cdbd7a03d918f with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash f4c78d6e3f105494bfbdd7585620a15c parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/40779ba5-7410-48b6-9925-fa82df90fd6a_characterizeImage_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/40779ba5-7410-48b6-9925-fa82df90fd6a_characterizeImage_1039672_19.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/269494ee-c109-4f39-ace9-e1d233a58351_isr_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/269494ee-c109-4f39-ace9-e1d233a58351_isr_1039672_19.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec81ffbad0c61be27833afa956380edd with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 657b55ba03a2c06d63e8464e851a47e7 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/94cc0521-e8c9-483c-bb2b-6de986065c30_characterizeImage_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/94cc0521-e8c9-483c-bb2b-6de986065c30_characterizeImage_1039769_50.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7268f3d7-e8c8-4892-b986-2df380e248a0_isr_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/7268f3d7-e8c8-4892-b986-2df380e248a0_isr_1039769_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:33 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:33 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:34 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:34 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:35 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:35 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b72cff528ab8f47382bb15f80348af48 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash b74d840327f4cad1a5ff43ce89e33de0 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b8b20da2-ba2f-4ea3-ac3b-ad0ceba67074_characterizeImage_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b8b20da2-ba2f-4ea3-ac3b-ad0ceba67074_characterizeImage_1039672_29.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/feeb7895-a082-4666-930e-34726bc052c7_isr_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/feeb7895-a082-4666-930e-34726bc052c7_isr_1039672_29.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e230f4942cde419d5c1c18b877ca7e4 with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 27918c062798519134fcd6a68c33bcc1 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/453e38c0-0bf6-429d-a902-19ebb161416c_characterizeImage_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/453e38c0-0bf6-429d-a902-19ebb161416c_characterizeImage_1039768_12.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1af7fcc2-a81e-41a1-8ee6-53af5f2d3923_isr_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/1af7fcc2-a81e-41a1-8ee6-53af5f2d3923_isr_1039768_12.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5655e5189a1afc94b51801998bd149d6 with result from task 906 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:36 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 642cbb134df64f0c7c50903cf68b9864 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/eb70b5c2-7ebe-486c-93a0-d8fd01399c0a_characterizeImage_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/eb70b5c2-7ebe-486c-93a0-d8fd01399c0a_characterizeImage_1039672_57.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/3644b0d3-f098-4047-97a1-716b67087bfd_isr_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/3644b0d3-f098-4047-97a1-716b67087bfd_isr_1039672_57.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee607042e2e72ca0578f256360181c13 with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash a7920f813ae7174fa5c48a98e124d8d3 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b9804e2f-8622-430f-ba67-416e1a44d801_characterizeImage_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b9804e2f-8622-430f-ba67-416e1a44d801_characterizeImage_1039673_56.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/27665831-35b2-4471-acd7-1d805a11e28f_isr_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/27665831-35b2-4471-acd7-1d805a11e28f_isr_1039673_56.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5cbab7542768eae2ac45d6d14ac4faa with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash f87c465cfee52e77018bfed640b9e862 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/73f26676-5f61-49de-8446-421730393e90_characterizeImage_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/73f26676-5f61-49de-8446-421730393e90_characterizeImage_1039769_26.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/37843739-27f9-4f00-9657-38b6e8da2614_isr_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/37843739-27f9-4f00-9657-38b6e8da2614_isr_1039769_26.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07a748e980b727917ccc2d2ccf00659d with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash e2f17e3dacbf86240600183bcd034cba parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/22fd8256-859a-4aa1-91b7-c1c3a269023e_characterizeImage_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/22fd8256-859a-4aa1-91b7-c1c3a269023e_characterizeImage_1039770_26.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e1b41f9e-010c-45fa-859c-6f2b2fc1162a_isr_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/e1b41f9e-010c-45fa-859c-6f2b2fc1162a_isr_1039770_26.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0197648506fb0213d140beec3243a8b with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 237c062ab715347acd8250a01ff39689 parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5004d6f8-a1c9-4794-94dd-c78f3e29d987_characterizeImage_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5004d6f8-a1c9-4794-94dd-c78f3e29d987_characterizeImage_1039673_62.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1e70e906-6058-440c-8801-1f7a257b55d0_isr_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1e70e906-6058-440c-8801-1f7a257b55d0_isr_1039673_62.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 627a22e664c178333f7fd79072c884c4 with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash b33a8fcc635f82d0c225a2082e3b1a06 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bd0c07ce-6030-44d5-b346-a427d392bd0a_characterizeImage_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bd0c07ce-6030-44d5-b346-a427d392bd0a_characterizeImage_1039768_41.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/236b6243-44b2-4ecf-8bea-59db07527fbf_isr_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/236b6243-44b2-4ecf-8bea-59db07527fbf_isr_1039768_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:37 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:37 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:38 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:39 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7d8f108aa9178eee03a0752fd305ccd with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 87af97eb0eb260a40f1f83c66d78de2d parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3ffaa2dd-2dab-44b7-98ed-4c54b8813270_characterizeImage_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3ffaa2dd-2dab-44b7-98ed-4c54b8813270_characterizeImage_1039769_4.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c9afc6de-8162-4be0-a727-ce3a003e273b_isr_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/c9afc6de-8162-4be0-a727-ce3a003e273b_isr_1039769_4.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5716012de01ac771e0a64ec45f62be38 with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash bedf7e7161dc1b92f830b308e7a27177 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/63af07f6-5bd0-4cb1-a6ed-4f0d48d2a032_characterizeImage_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/63af07f6-5bd0-4cb1-a6ed-4f0d48d2a032_characterizeImage_1039674_18.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c506cf56-2725-468c-ba9b-6c650a2109d7_isr_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/c506cf56-2725-468c-ba9b-6c650a2109d7_isr_1039674_18.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cf948f84808a05ac7a3fdbdda775c82 with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 2a6fc7999f00f203c21c9c0eb00e904d parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1a39406a-dc14-4b28-bfb5-aa92e55ac7ed_characterizeImage_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1a39406a-dc14-4b28-bfb5-aa92e55ac7ed_characterizeImage_1039672_33.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cc3cabc1-a8a7-47c8-ba9a-a5abbcb617e3_isr_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/cc3cabc1-a8a7-47c8-ba9a-a5abbcb617e3_isr_1039672_33.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00716c9a7dd060f0c8347dc89efd1424 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash e5c15992dd5ea8834d5e2c28d05c34bf parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/699358c1-8228-4c6a-84ee-19cd5a779c62_characterizeImage_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/699358c1-8228-4c6a-84ee-19cd5a779c62_characterizeImage_1039770_39.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0434aa7a-de16-4aa1-bc93-227cbc70516d_isr_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/0434aa7a-de16-4aa1-bc93-227cbc70516d_isr_1039770_39.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 499e1c78784881617c3ecfa036174c53 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 6fb14b01f918eda680180fe48e1156a4 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0a5d8a0e-4ba8-4fa9-94b0-6ae11d99119c_isr_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0a5d8a0e-4ba8-4fa9-94b0-6ae11d99119c_isr_1039769_2.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a40c8719a275b090816d5745f51044c with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 02f8f59cc9323f5bf6c625469d7b345b parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/daf7400f-7678-447d-b769-4400811b2541_characterizeImage_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/daf7400f-7678-447d-b769-4400811b2541_characterizeImage_1039769_38.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/565cb7d4-3066-4006-8b89-1399cc8ea6a2_isr_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/565cb7d4-3066-4006-8b89-1399cc8ea6a2_isr_1039769_38.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7953789804f2bbf536a3492562c051fa with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 162853302feb852a8ac8c011c4f12748 parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3b0c1ae3-9882-47f0-b3d1-47e5023d126d_characterizeImage_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3b0c1ae3-9882-47f0-b3d1-47e5023d126d_characterizeImage_1039672_31.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/54edcf56-2ac6-46b2-bc1c-3f970c2b0129_isr_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/54edcf56-2ac6-46b2-bc1c-3f970c2b0129_isr_1039672_31.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb3a668eebcbb50de018e97fee0bd5a2 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash ef9a4f83390781f3187419609f53cd38 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da8293d7-6cc7-40e9-982c-fbf265565b42_characterizeImage_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da8293d7-6cc7-40e9-982c-fbf265565b42_characterizeImage_1039674_3.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4217ed90-0996-4ac6-bf56-a188159cc873_isr_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4217ed90-0996-4ac6-bf56-a188159cc873_isr_1039674_3.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f63197f09c835643cc6bf3db562f47 with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash 45df89ed83d60cca918ca73c8127fab1 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/aec25512-d293-4b7c-9185-56be329015c7_characterizeImage_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/aec25512-d293-4b7c-9185-56be329015c7_characterizeImage_1039673_47.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7405b285-f53a-4386-ab3a-7090b23482a9_isr_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7405b285-f53a-4386-ab3a-7090b23482a9_isr_1039673_47.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9983cea0c12b9fe1f0135e631feb1592 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 4511b07eec5784efe873cb636aaa953c parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/00b32d24-1148-4c1d-bae1-245642f3154d_characterizeImage_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/00b32d24-1148-4c1d-bae1-245642f3154d_characterizeImage_1039770_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fe9741f8-7784-499b-81ca-22624fea81e5_isr_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/fe9741f8-7784-499b-81ca-22624fea81e5_isr_1039770_8.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a187aa4d5cfad55638035b383a7dadf with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 1f332c215eb4db260f01bcd412514f4e parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6b07db35-e324-4964-b05d-4f07f2dc0ca9_characterizeImage_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6b07db35-e324-4964-b05d-4f07f2dc0ca9_characterizeImage_1039674_45.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4f3beb2a-50c4-4384-88f4-b7dd5a0732c7_isr_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4f3beb2a-50c4-4384-88f4-b7dd5a0732c7_isr_1039674_45.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 061274326a9ed7693643872e02bff783 with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash 56bfbc54d02147538e4f2b926600c342 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c660a4ee-734f-415a-9209-9ff718735bb4_characterizeImage_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c660a4ee-734f-415a-9209-9ff718735bb4_characterizeImage_1039673_23.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/056e18dd-25b1-4d4a-a354-c46b3f548450_isr_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/056e18dd-25b1-4d4a-a354-c46b3f548450_isr_1039673_23.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68078498264dad068b0e420f51d16b2 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash 874f86bdc1ca4b64935cd575138435ef parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/129b02d3-f9e0-481b-9963-17905ecffe12_characterizeImage_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/129b02d3-f9e0-481b-9963-17905ecffe12_characterizeImage_1039769_12.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0d9f564f-0c5c-4e42-9087-779547974c6f_isr_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/0d9f564f-0c5c-4e42-9087-779547974c6f_isr_1039769_12.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae7b7b171d08cbd592f0de95094ad763 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash 90508cbcf6904e8924ee852b8be721be parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/35f3e69c-ea30-493d-a9c4-2adee87c86bc_characterizeImage_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/35f3e69c-ea30-493d-a9c4-2adee87c86bc_characterizeImage_1039768_21.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e856f87b-dd66-451f-aa09-07d2bb5c0d58_isr_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/e856f87b-dd66-451f-aa09-07d2bb5c0d58_isr_1039768_21.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47b8056bfa3e79a288381b825b81f9cb with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash f25d939a11722cf977bb6a5c65febf69 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e172a2e-1003-4d4f-8de4-d0b4bcdf3a96_characterizeImage_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e172a2e-1003-4d4f-8de4-d0b4bcdf3a96_characterizeImage_1039672_5.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8320ecc7-1d39-45a4-a06f-f6430653717c_isr_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8320ecc7-1d39-45a4-a06f-f6430653717c_isr_1039672_5.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b62409ba4f0ebd936a42612348c787 with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 6a7914c091db6e8a09afff4802ceffa2 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/91752812-65d8-49d9-b57f-86bbd89bdc97_characterizeImage_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/91752812-65d8-49d9-b57f-86bbd89bdc97_characterizeImage_1039768_9.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/a51e1f1c-1683-4289-86d5-b5535617aa86_isr_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/a51e1f1c-1683-4289-86d5-b5535617aa86_isr_1039768_9.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d6290122990ace408b008297d6b7acc with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 89648522eb29978e5e10cba75bbd51d3 parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdad1e69-59fe-4010-972d-656398c7809c_characterizeImage_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdad1e69-59fe-4010-972d-656398c7809c_characterizeImage_1039768_19.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/cb3516fc-3924-4db3-b715-60b84c20f8b1_isr_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/cb3516fc-3924-4db3-b715-60b84c20f8b1_isr_1039768_19.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16ea490e7d544558c8138fa220c99548 with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 1430c5af4f15b9881faa7fa62830b668 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a0ad58e-0ce3-4ef0-a98a-0b672096ba8f_characterizeImage_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a0ad58e-0ce3-4ef0-a98a-0b672096ba8f_characterizeImage_1039768_39.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/34c7bbf4-23f0-46e6-93cb-53db66425f1e_isr_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/34c7bbf4-23f0-46e6-93cb-53db66425f1e_isr_1039768_39.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdf75693cad8965b3591f16f378f8896 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash e90c2f67b5d0e9cdcf93669cfa5cfeb8 parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a0750179-4c89-4978-8e0c-472b80d155aa_characterizeImage_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a0750179-4c89-4978-8e0c-472b80d155aa_characterizeImage_1039770_13.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/8b5e51c2-5a06-4a9a-8860-3bcd739a395c_isr_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/8b5e51c2-5a06-4a9a-8860-3bcd739a395c_isr_1039770_13.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d38263dcaa394a87ccd02d3a87ae6c7 with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash b9595382aef5c5d99c8270a25b52f2e3 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/55a4c813-1264-4e1d-a1dd-5a6badfa273e_characterizeImage_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/55a4c813-1264-4e1d-a1dd-5a6badfa273e_characterizeImage_1039672_25.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ef31bc9d-a4a4-4bdb-b30f-a6c417d4ff39_isr_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/ef31bc9d-a4a4-4bdb-b30f-a6c417d4ff39_isr_1039672_25.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54d2e79f3be85527cea1d0fce74eeedd with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash f01a9bb8e23eff4598d8ffb7f2d3a781 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/200a6d68-4733-4d9b-a9f4-1f772e7c7d3b_characterizeImage_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/200a6d68-4733-4d9b-a9f4-1f772e7c7d3b_characterizeImage_1039674_11.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f3ee55bc-d2b4-40ec-b915-07ee069e2719_isr_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/f3ee55bc-d2b4-40ec-b915-07ee069e2719_isr_1039674_11.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcf9528c3ae0dc7be11c6e62c913bbe8 with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash c3db68f8f0e62a608118fd2f68d40130 parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/033d07da-e218-46f4-b1d1-c8460d233f4e_characterizeImage_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/033d07da-e218-46f4-b1d1-c8460d233f4e_characterizeImage_1039672_28.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/57706252-3ead-439a-9a25-6cebc4e101f7_isr_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/57706252-3ead-439a-9a25-6cebc4e101f7_isr_1039672_28.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20fd64f63aaf46ba99586d38f32347db with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash 0d421a6930aacbb5d0de7cb6eb4b87c7 parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/38d1743e-6422-4bb9-bebb-3beef95ad9fc_characterizeImage_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/38d1743e-6422-4bb9-bebb-3beef95ad9fc_characterizeImage_1039770_14.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1c5d3d3d-52a7-4077-8b84-8f929290e302_isr_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/1c5d3d3d-52a7-4077-8b84-8f929290e302_isr_1039770_14.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b93a2eddb45671f2a8e82393195779c with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash b2dfb8d833966e1406004313d51b931d parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/238f7a5c-c45e-439a-b82d-94024ca9b3bf_characterizeImage_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/238f7a5c-c45e-439a-b82d-94024ca9b3bf_characterizeImage_1039672_34.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/49c44b1e-8cb6-4bcf-943f-a9affcaa85a3_isr_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/49c44b1e-8cb6-4bcf-943f-a9affcaa85a3_isr_1039672_34.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de3ad4ed56ecaa3950a9be383d665724 with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash 00aef9c3ae64416d510f70bed59147ab parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e2ae242a-59ea-4f72-a28e-9531b264d2c6_characterizeImage_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e2ae242a-59ea-4f72-a28e-9531b264d2c6_characterizeImage_1039769_60.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/277f1ef7-d21b-4b6d-b0a1-09effb22c0b9_isr_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/277f1ef7-d21b-4b6d-b0a1-09effb22c0b9_isr_1039769_60.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce006f857edf040775aeaef5e049bd38 with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash f402ea8a8d5f07f84f588e0e3aa220eb parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b070a01f-05a9-4f40-8ff5-bbd41eb296d1_characterizeImage_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b070a01f-05a9-4f40-8ff5-bbd41eb296d1_characterizeImage_1039672_11.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8a6def3e-7943-478f-af3f-4d678eef3a13_isr_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/8a6def3e-7943-478f-af3f-4d678eef3a13_isr_1039672_11.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba305b3d41fd87aef9cea0e9bf7cb14d with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 7a90069540e6585b69ae81549b8e60ea parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5b180f17-3673-4cfe-a7e5-c9698babd0ba_characterizeImage_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5b180f17-3673-4cfe-a7e5-c9698babd0ba_characterizeImage_1039674_41.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aad4d5c5-5ea1-49eb-8608-85c00b2ae714_isr_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aad4d5c5-5ea1-49eb-8608-85c00b2ae714_isr_1039674_41.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1029b13f004d1993d83c6ac3363965f9 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 202c1b3b89a1224cc165b9739013de15 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6a7a2f4c-5cd1-4ef9-9ec6-d780821a1d6f_characterizeImage_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6a7a2f4c-5cd1-4ef9-9ec6-d780821a1d6f_characterizeImage_1039673_61.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ebdf5f65-aea1-4d23-a4a5-abd0e280fad0_isr_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/ebdf5f65-aea1-4d23-a4a5-abd0e280fad0_isr_1039673_61.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f093cb047cceb1c191fc799a7d60a2 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash b843d5aa1d3483b8f949647550c76e63 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/9795f375-835b-43de-85ba-5019c85c6bdb_characterizeImage_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/9795f375-835b-43de-85ba-5019c85c6bdb_characterizeImage_1039768_56.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/0806d547-b491-45be-9893-0cb17c82463f_isr_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/0806d547-b491-45be-9893-0cb17c82463f_isr_1039768_56.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c2eeb3c843816d9dadfd062be43f7b1 with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 8e1b549244db664f7e78733e22142268 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d9ac676-b90a-4e07-a694-b721eacb8099_characterizeImage_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d9ac676-b90a-4e07-a694-b721eacb8099_characterizeImage_1039769_51.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/2503fd28-0233-4b8e-b6ec-6e9828efba3e_isr_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/2503fd28-0233-4b8e-b6ec-6e9828efba3e_isr_1039769_51.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 515d4211514b2056e9ff758d43af24e5 with result from task 945 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:40 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 2ac7bf9d62aacf1cc46cba355f34f532 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/06836a71-92e3-47ea-96b0-d77bb61b78ca_characterizeImage_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/06836a71-92e3-47ea-96b0-d77bb61b78ca_characterizeImage_1039769_3.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bcea016d-47f6-4546-a0b9-9c947c8c1432_isr_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bcea016d-47f6-4546-a0b9-9c947c8c1432_isr_1039769_3.stderr parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571fbcec90aea84d5435d748219c21fb with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash ad2937afc2f697b335307dadb878fc5f parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/80ae5e2b-51bb-463d-9b76-06194f37d592_characterizeImage_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/80ae5e2b-51bb-463d-9b76-06194f37d592_characterizeImage_1039672_1.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b49e8b96-94e7-4563-be48-0049e7c098dd_isr_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/b49e8b96-94e7-4563-be48-0049e7c098dd_isr_1039672_1.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53f6679eaa733f46a93115aabde24d5 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 1574cbb08a4adc8d505549bf46281724 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3f7db7ff-3eaf-4b60-992f-fe26fc3a10ee_characterizeImage_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3f7db7ff-3eaf-4b60-992f-fe26fc3a10ee_characterizeImage_1039672_54.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/31025ffc-d2b0-45b9-ad3a-d39f35127e51_isr_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/31025ffc-d2b0-45b9-ad3a-d39f35127e51_isr_1039672_54.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc582eaf7d6d7443fc6099f0ad7c77a with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 0a87367264c9de814e82919d3ef68759 parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0129c161-1a6d-470d-8482-6b46247da61a_characterizeImage_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0129c161-1a6d-470d-8482-6b46247da61a_characterizeImage_1039770_16.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7f16bf58-2be1-4f0b-b481-b024d2e5e3bb_isr_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/7f16bf58-2be1-4f0b-b481-b024d2e5e3bb_isr_1039770_16.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fadaeee92e21d777df0022991c732c9e with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 0c4aeec3f772e364986b7d82c9fefda7 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b1c428a8-98fa-4120-8636-1029d63dacff_characterizeImage_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b1c428a8-98fa-4120-8636-1029d63dacff_characterizeImage_1039674_26.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4512c4da-d9db-49ce-9b99-f94cef217c7c_isr_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/4512c4da-d9db-49ce-9b99-f94cef217c7c_isr_1039674_26.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 362ba3ad71f23e97ca63f380e4732f67 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 7fe24c89116654ffcd85e7356458f030 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c0b4bbb8-e132-4a27-ab74-070c6ec0fad4_characterizeImage_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c0b4bbb8-e132-4a27-ab74-070c6ec0fad4_characterizeImage_1039769_32.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3e47d515-0bbf-42a1-b734-0f35fd7abdce_isr_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/3e47d515-0bbf-42a1-b734-0f35fd7abdce_isr_1039769_32.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f55a880acb2468a7cd3eb2e8404a7274 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash cc05b6b974dfa702b71d0b73264d79d4 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/ea323feb-31fb-474b-863e-61fbafe103e4_characterizeImage_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/ea323feb-31fb-474b-863e-61fbafe103e4_characterizeImage_1039673_4.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3d665c1-a973-4fee-91fe-75777420339e_isr_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/f3d665c1-a973-4fee-91fe-75777420339e_isr_1039673_4.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5df2399602351e9cafe088e640172da with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash eb00b0f71d3610bd258e7a9052971b15 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3809a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3879b6d-7d80-4737-be5c-f514d4399162_characterizeImage_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3879b6d-7d80-4737-be5c-f514d4399162_characterizeImage_1039769_27.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/85cfbba5-269f-40e5-aaa6-4c6d17ba0402_isr_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/85cfbba5-269f-40e5-aaa6-4c6d17ba0402_isr_1039769_27.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a96835c18c8dd6901c49a6ecd6a0d2d0 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 019d3df643bba435ca97c3da4a58d1dc parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0cd0be50-fa26-41e4-9fbf-41affb874cca_characterizeImage_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0cd0be50-fa26-41e4-9fbf-41affb874cca_characterizeImage_1039770_24.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/50d19a15-b33b-452d-935a-3be7a6d1b1d2_isr_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/50d19a15-b33b-452d-935a-3be7a6d1b1d2_isr_1039770_24.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a22d2a169eb3693a704ca2a2757c6cb6 with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 8207795c46eddee15b784bc468470cfa parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a6aef048-b551-4641-983a-424224731ff4_characterizeImage_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a6aef048-b551-4641-983a-424224731ff4_characterizeImage_1039673_20.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3df7d690-1e64-42e6-aaf1-6da71af4c3e3_isr_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3df7d690-1e64-42e6-aaf1-6da71af4c3e3_isr_1039673_20.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7229e00df6b425e89b2e6f1754ccb3 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash c7628a627fc0246e2bcaea1f6edcf45d parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3e15da0f-a1bd-4d81-8fb4-5662b522d70a_characterizeImage_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3e15da0f-a1bd-4d81-8fb4-5662b522d70a_characterizeImage_1039673_8.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a1641c1b-68e2-42b6-b941-c0364d475cfd_isr_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a1641c1b-68e2-42b6-b941-c0364d475cfd_isr_1039673_8.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a019b719a2e9e5c31248f8d4023358 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 12ea6cab135f276c3ae1e27d81dde56a parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c5e3f89-289b-40c6-a3a4-2770e0e4d2bd_characterizeImage_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c5e3f89-289b-40c6-a3a4-2770e0e4d2bd_characterizeImage_1039673_43.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0a428ed0-77ac-4825-8581-ed0771abb165_isr_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/0a428ed0-77ac-4825-8581-ed0771abb165_isr_1039673_43.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbd61c3b3203e43a5d3965c72a5d066c with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 4c1793c1a8271af7ed7cde549bc5961b parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a381b361-50ef-4f46-a91c-25c07b473188_characterizeImage_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a381b361-50ef-4f46-a91c-25c07b473188_characterizeImage_1039770_46.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a73c514b-6c4e-4703-9a5d-ba41be403a5d_isr_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/a73c514b-6c4e-4703-9a5d-ba41be403a5d_isr_1039770_46.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db228d7ed1b867014e46f994dd6f8af8 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 07c0699b1f45c0676011935555571cd9 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/012f55c1-b670-4549-878e-b5397649d8c1_characterizeImage_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/012f55c1-b670-4549-878e-b5397649d8c1_characterizeImage_1039674_51.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8712a2ae-27fd-4520-b3bd-b9eea3b4f1a6_isr_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/8712a2ae-27fd-4520-b3bd-b9eea3b4f1a6_isr_1039674_51.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4416ff466e3fa7a5481aacee246b09d with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 6848d233f8db78643880f285dc151f2b parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/714d1cd7-42b3-4b6c-94e3-3001262f1b66_characterizeImage_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/714d1cd7-42b3-4b6c-94e3-3001262f1b66_characterizeImage_1039672_18.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64a3c2a5-99a7-4d72-b2a8-2584dbd5f762_isr_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/64a3c2a5-99a7-4d72-b2a8-2584dbd5f762_isr_1039672_18.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5aa0e03300468dc82a082a2314b1c7e with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash d60dda7aead96399b1730b80ba9d1c06 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/12a683a3-de3f-4d29-ac3a-5973f45406bd_characterizeImage_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/12a683a3-de3f-4d29-ac3a-5973f45406bd_characterizeImage_1039768_33.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/57b22b56-3f3d-483a-a3d4-0d05ee67a11e_isr_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/57b22b56-3f3d-483a-a3d4-0d05ee67a11e_isr_1039768_33.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e860d74e054c34d0e3541c78015bd36e with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 105b40d176936c9b538a5a17be8835a4 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54397817-d736-44a2-89fa-85848addbacf_characterizeImage_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54397817-d736-44a2-89fa-85848addbacf_characterizeImage_1039770_37.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c241f33e-3c4d-4ada-9c7e-ba108abaa436_isr_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c241f33e-3c4d-4ada-9c7e-ba108abaa436_isr_1039770_37.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73f25e6755b21e3a03cf7114d58d30d3 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 5124bc0b1ce59bf73a519b271dc17226 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/9f1255a8-a7a7-4762-a226-af6343e1f403_characterizeImage_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/9f1255a8-a7a7-4762-a226-af6343e1f403_characterizeImage_1039769_10.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/927b506b-0af5-4ce0-b59e-9a4eb53bd417_isr_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/927b506b-0af5-4ce0-b59e-9a4eb53bd417_isr_1039769_10.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 064e8120a2dabfc571328b1f71462789 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 8bac1841fc1661b25ef598eea8c3a0ad parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5238ee21-b584-45d0-a0a5-59290cfdad2f_characterizeImage_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5238ee21-b584-45d0-a0a5-59290cfdad2f_characterizeImage_1039674_57.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/67114727-7d7d-4bb2-9649-2696703bfd24_isr_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/67114727-7d7d-4bb2-9649-2696703bfd24_isr_1039674_57.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c921b4648e485ad0b5f7ed3798766d92 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 2ed1d2147301547cf6470477aa5f8bfe parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9e5a41da-d991-42ef-a923-b4cdf7c53559_characterizeImage_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9e5a41da-d991-42ef-a923-b4cdf7c53559_characterizeImage_1039673_33.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1951b5ed-cf68-4f99-a9bd-16e83c2265f0_isr_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/1951b5ed-cf68-4f99-a9bd-16e83c2265f0_isr_1039673_33.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19e9eb3ea984e0432634d36afa76fed1 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 91656d3fbe253cf9b8bfc431a7abee63 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b33c9068-d7db-4615-85eb-2d96888a07c8_characterizeImage_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b33c9068-d7db-4615-85eb-2d96888a07c8_characterizeImage_1039769_41.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/253c6072-4d13-4841-b8f5-d911ef28e904_isr_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/253c6072-4d13-4841-b8f5-d911ef28e904_isr_1039769_41.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e490d38f09efebbbf3803501dedffe53 with result from task 420 parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:40 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash b23ade52f300207ca67507424ca668b1 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/275425f6-759e-41ce-99ac-dbe453c9fc60_characterizeImage_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/275425f6-759e-41ce-99ac-dbe453c9fc60_characterizeImage_1039673_45.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/144a5c82-48b1-4557-be48-511347e8a07a_isr_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/144a5c82-48b1-4557-be48-511347e8a07a_isr_1039673_45.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ef2f33ff9aea03102d311e284adf44f with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 34edd86224df89cb1b2ee90062c27146 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/66962144-03ce-45e3-8f5f-a1388231a522_characterizeImage_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/66962144-03ce-45e3-8f5f-a1388231a522_characterizeImage_1039673_46.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7cc318e3-25ca-47fb-8150-ba76cd64f2ff_isr_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/7cc318e3-25ca-47fb-8150-ba76cd64f2ff_isr_1039673_46.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a88bec5d30e778d83e7b35b9c59b3753 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 96462759816b30abb2afd2921f5a4762 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b90ba4e6-f991-4f03-b5af-55c8724ec8c3_characterizeImage_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b90ba4e6-f991-4f03-b5af-55c8724ec8c3_characterizeImage_1039672_21.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/e0723edd-3904-4a7f-a461-6853643f37f5_isr_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/e0723edd-3904-4a7f-a461-6853643f37f5_isr_1039672_21.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9688632636a02c379135aad5fd25919 with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 0e202f99f2016a9a5d839d92a0870951 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5d0e85e1-cd8c-471b-a6cf-e1d51e88cfec_characterizeImage_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5d0e85e1-cd8c-471b-a6cf-e1d51e88cfec_characterizeImage_1039674_39.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cbe4a2bf-3c64-41c2-90cb-228b7e8a704a_isr_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/cbe4a2bf-3c64-41c2-90cb-228b7e8a704a_isr_1039674_39.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12b94c55db4ec5e204453ae563a84bff with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 54a3926123ee5b04806b755fdc395d7f parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3813a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/540be8eb-3066-4ae7-b3e6-9cc910939432_characterizeImage_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/540be8eb-3066-4ae7-b3e6-9cc910939432_characterizeImage_1039768_40.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9454bbc1-f149-4bfb-a22a-b6de74b2cb40_isr_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/9454bbc1-f149-4bfb-a22a-b6de74b2cb40_isr_1039768_40.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9da059e62762928316e05f1793f1c9c with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash 90f6154bd3671c6404dad7fe1e1b2b48 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/7b497ac1-15ce-4f24-b47d-169727dc43fe_characterizeImage_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/7b497ac1-15ce-4f24-b47d-169727dc43fe_characterizeImage_1039770_57.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/95d5cb3e-256c-4931-898f-1bb20c659156_isr_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/95d5cb3e-256c-4931-898f-1bb20c659156_isr_1039770_57.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27ca97a11d1309f5d2131570fdf133ff with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash ffc5be0706adec58282a39376ec36748 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b716fef5-5fa9-4fdd-ad80-f3d82ae7dd42_characterizeImage_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b716fef5-5fa9-4fdd-ad80-f3d82ae7dd42_characterizeImage_1039770_56.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2019cf48-ca29-49aa-b22a-78d917218d69_isr_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/2019cf48-ca29-49aa-b22a-78d917218d69_isr_1039770_56.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63b0edd5ad252ae7c1a91650ab3adda0 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash 165036fa858d242fbf2e2335396c5c1c parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d91e192f-aae9-4f6c-92b1-638d611c282d_characterizeImage_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d91e192f-aae9-4f6c-92b1-638d611c282d_characterizeImage_1039769_21.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1a970642-948c-4634-8c9f-2ca549e87987_isr_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1a970642-948c-4634-8c9f-2ca549e87987_isr_1039769_21.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa5e39136bf40842c7ffa9a3e6e3f131 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 7e7f0e5328f8ffb235c435be54b8a2a7 parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/69f517e6-c5b9-4d8b-b1f5-6a4031d3dcf9_characterizeImage_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/69f517e6-c5b9-4d8b-b1f5-6a4031d3dcf9_characterizeImage_1039768_26.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d461f5f4-a7ae-4f3c-8643-0a5bd23b391a_isr_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d461f5f4-a7ae-4f3c-8643-0a5bd23b391a_isr_1039768_26.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a78a99806b8f5932157c8e79aaf1d77b with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 34feebcea330997127a8d0b838bf6c16 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e26b1815-91bc-44ad-893a-9261a7f8c04a_characterizeImage_1039674_50.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e26b1815-91bc-44ad-893a-9261a7f8c04a_characterizeImage_1039674_50.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aef2db73-c7a4-4d7a-803c-a857aea47f55_isr_1039674_50.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/aef2db73-c7a4-4d7a-803c-a857aea47f55_isr_1039674_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 217 try 1 failed parsl.dataflow.dflow ERROR: Task 217 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f5e836b0f93fee4fad4f087b7911a2d5 with result from task 217 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 218 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 218 try 0 failed parsl.dataflow.dflow INFO: Task 218 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/384c0dae-ce3b-4a1d-beeb-260b99e95465_calibrate_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/384c0dae-ce3b-4a1d-beeb-260b99e95465_calibrate_1039672_2.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ca1b93f-d54e-4fcb-ad49-77e31433a39c_characterizeImage_1039672_2.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5924b507c809b8efe9ffa842fcd7491a with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash b1ff7e234c225c63b86be1707a70695e parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/adf4f691-5656-4077-a9bc-dd807989def9_characterizeImage_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/adf4f691-5656-4077-a9bc-dd807989def9_characterizeImage_1039673_5.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d500ff7-0815-4968-9048-51cd60b83785_isr_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/2d500ff7-0815-4968-9048-51cd60b83785_isr_1039673_5.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef6d1a4369d12d0daaa6cb6f83fb79c2 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash 2e16e244996602ebce5dc190292f022c parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/fecd1772-cf58-4303-a047-7a24348c051c_characterizeImage_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/fecd1772-cf58-4303-a047-7a24348c051c_characterizeImage_1039769_33.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bb8017ce-1432-4169-bce4-166d5b2cffd5_isr_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/bb8017ce-1432-4169-bce4-166d5b2cffd5_isr_1039769_33.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aad2e7f869adea2f8bafbb07bb2025fc with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash 04f9dc0cf3c025d8b2293c298bf98cd0 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/46e07f1b-f925-4afc-a03f-1b8de454c0e5_characterizeImage_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/46e07f1b-f925-4afc-a03f-1b8de454c0e5_characterizeImage_1039673_28.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a27885a2-e3cd-41bf-8233-7d373676e0b2_isr_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/a27885a2-e3cd-41bf-8233-7d373676e0b2_isr_1039673_28.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 536cb5292176c1db8ac6a3e205f885f3 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 8a71d5b9eb269b14cddd3eaa479a0259 parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/77221497-5df3-495f-bc6c-8bc6674d1e9e_characterizeImage_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/77221497-5df3-495f-bc6c-8bc6674d1e9e_characterizeImage_1039672_53.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a2825217-ad35-4e11-a2ec-893846cfc618_isr_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/a2825217-ad35-4e11-a2ec-893846cfc618_isr_1039672_53.stderr parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df5b6ad455531f74a5ee8e3b53d7468b with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash fb11a0548fda0d1bdee3c110a760b43d parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8e3bcfbb-1ea5-473b-870c-8d205ceba8a8_characterizeImage_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8e3bcfbb-1ea5-473b-870c-8d205ceba8a8_characterizeImage_1039673_30.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3c46d659-7384-4cb2-8a24-5ad0d8805d39_isr_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/3c46d659-7384-4cb2-8a24-5ad0d8805d39_isr_1039673_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:41 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:41 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e6d65fd23b1680158800aa91cf7811c with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 7e681d233c2232bc0cb1504df02f81a9 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bc2c99fe-acc0-46e1-8954-5f186793039f_characterizeImage_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bc2c99fe-acc0-46e1-8954-5f186793039f_characterizeImage_1039770_43.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/26960b24-44df-42a9-8c2e-a743b70568f9_isr_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/26960b24-44df-42a9-8c2e-a743b70568f9_isr_1039770_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:41 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:42 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:43 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0842b2d58fc7679bd038a29e4279a51 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash e6e80ae3691e147dfab084f9d1b92cc6 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/60010f75-8990-4cb1-aa49-0f701ac6ba91_characterizeImage_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/60010f75-8990-4cb1-aa49-0f701ac6ba91_characterizeImage_1039769_62.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f22eb894-5dce-48a2-9ba7-a20e72a05da2_isr_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/f22eb894-5dce-48a2-9ba7-a20e72a05da2_isr_1039769_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:43 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:44 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:44 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad1a46f0bcd7b44a8e53856296c4bb77 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash b33c8e30b105fb0004782cd5ef41f604 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/ad9a6a88-2436-448d-bc2a-e4c4b4586a13_characterizeImage_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/ad9a6a88-2436-448d-bc2a-e4c4b4586a13_characterizeImage_1039674_62.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1e607320-51de-4af5-a9c9-bf0d9a5cb390_isr_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/1e607320-51de-4af5-a9c9-bf0d9a5cb390_isr_1039674_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:44 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7631f3c52bd227c4e59837bc619ca758 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash bd152039c3ec59d467f40c1a4fe6a632 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/255e7c21-ca5c-46c3-8e7e-473ffbb505cb_characterizeImage_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/255e7c21-ca5c-46c3-8e7e-473ffbb505cb_characterizeImage_1039673_55.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/19897253-7d6a-4b4c-8573-4ea792f17c73_isr_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/19897253-7d6a-4b4c-8573-4ea792f17c73_isr_1039673_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:45 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867300780258723a0ddb4ab1f5ab64de with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 4484e12fc73290f25303f96fadc9e2c6 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/bcc9c4a3-8ba5-4ba9-88a9-1601d6a20627_characterizeImage_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/bcc9c4a3-8ba5-4ba9-88a9-1601d6a20627_characterizeImage_1039769_57.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1daff6da-2e2e-48e7-95f6-bbbfec85dbf0_isr_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/1daff6da-2e2e-48e7-95f6-bbbfec85dbf0_isr_1039769_57.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9406cce65eef05d3c6bb7548b0789eb2 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 257ed837650ffe714bd3278b7fb466ae parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f86d8ade-84a3-454e-b0f8-92f6c2dfea23_calibrate_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f86d8ade-84a3-454e-b0f8-92f6c2dfea23_calibrate_1039673_32.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f779bcd-99e6-4242-93eb-2eb209fa1613_characterizeImage_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f779bcd-99e6-4242-93eb-2eb209fa1613_characterizeImage_1039673_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:46 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6506b1b11aa7d5b12e969a2ed5a138ee with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 9c51adff1bbf125d600e41e91a1d6f0e parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2147c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/41dfe033-0c3a-47ae-991f-0ca2510f0555_characterizeImage_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/41dfe033-0c3a-47ae-991f-0ca2510f0555_characterizeImage_1039673_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/69f91105-4c3d-4fc5-aab4-8dc12d66d0f8_isr_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039673/69f91105-4c3d-4fc5-aab4-8dc12d66d0f8_isr_1039673_13.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5295a6d5dc5b3fc448c890aeab68c838 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 6a416b4087d87aebe7322ae0c6410739 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/524ce03c-fe72-48b6-ad56-dada9b1cfb5f_characterizeImage_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/524ce03c-fe72-48b6-ad56-dada9b1cfb5f_characterizeImage_1039672_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7a375cfb-3ec6-4e16-95a4-a0e79305c6a1_isr_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039672/7a375cfb-3ec6-4e16-95a4-a0e79305c6a1_isr_1039672_61.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 822a9801eefe2dea61f5a6f581f8f1f3 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash 691033662c472b9c09790bf85edc8867 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2144a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/34f51c13-db51-44a3-9255-281c8d5117e7_characterizeImage_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/34f51c13-db51-44a3-9255-281c8d5117e7_characterizeImage_1039769_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dabb7daf-ccdc-45bc-829d-39dbe0bf9128_isr_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/dabb7daf-ccdc-45bc-829d-39dbe0bf9128_isr_1039769_31.stderr parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64ef55a962b2662572b0f862656e66ac with result from task 1047 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 7a9daaf6464e0b42cf673a09e9315d0a parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2159e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ebce88-52e0-4be4-a2ab-bf272f266faf_characterizeImage_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ebce88-52e0-4be4-a2ab-bf272f266faf_characterizeImage_1039770_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c2e8544b-81df-416a-92f9-903e118ab3de_isr_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/c2e8544b-81df-416a-92f9-903e118ab3de_isr_1039770_45.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2efd2cbf426412ceaf094bd20e67f60 with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash c178e60c53e5ef13086774732179b260 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2156c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/16a616f3-f2a4-4c62-b02d-c27cdc449e3f_characterizeImage_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/16a616f3-f2a4-4c62-b02d-c27cdc449e3f_characterizeImage_1039674_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/6cb39efc-72a0-4b9c-90f2-68d05aadba5e_isr_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039674/6cb39efc-72a0-4b9c-90f2-68d05aadba5e_isr_1039674_56.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0d32ddff10fc32158dfed739ed786ab with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash cdc8528bd4c25b57bf78a50db2b4f489 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e8daf669-4aef-4285-b55a-4f63d8ecf040_characterizeImage_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e8daf669-4aef-4285-b55a-4f63d8ecf040_characterizeImage_1039768_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d7cd9e78-719b-47ab-80d6-971d2a6db9a6_isr_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/d7cd9e78-719b-47ab-80d6-971d2a6db9a6_isr_1039768_1.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a80151df9315fa71c746cc0aca316c with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash 196d2b1967c4a75bb922ac89241f6e4f parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a38cb0-c317-4ef4-af17-9a8710af5cf4_characterizeImage_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a38cb0-c317-4ef4-af17-9a8710af5cf4_characterizeImage_1039769_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/405621de-3c47-4624-b5cf-74fbafa6de30_isr_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039769/405621de-3c47-4624-b5cf-74fbafa6de30_isr_1039769_17.stderr parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1906a808b5abe1eb91d169de37b040d with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash a4120e8fe1d7f81969a967f9959e4ac4 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a92bc859-02c0-4859-9f59-0b6e551f7bbe_characterizeImage_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a92bc859-02c0-4859-9f59-0b6e551f7bbe_characterizeImage_1039770_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b9f4a176-ffe9-479f-877a-dbd6cc979208_isr_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039770/b9f4a176-ffe9-479f-877a-dbd6cc979208_isr_1039770_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:46 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:46 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:47 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:47 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:48 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08ad7b500f226ece234d0d5a41e16600 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 9af800eebbcaa6b11a270ac2e2f61fd9 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/256e0038-370b-464f-9f1c-b533b9903a1d_characterizeImage_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/256e0038-370b-464f-9f1c-b533b9903a1d_characterizeImage_1039768_48.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b3e11ec9-b539-47cf-af52-c4c1b908fc8e_isr_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/isr/1039768/b3e11ec9-b539-47cf-af52-c4c1b908fc8e_isr_1039768_48.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf629c92a3e90ffc48af473dbd8cb54f with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 377e6449ac1c26728f3937a2ba5b9fd6 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/3b518a7c-2590-4add-8cb2-b0054dd0d31f_calibrate_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/3b518a7c-2590-4add-8cb2-b0054dd0d31f_calibrate_1039674_40.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93787240-5f77-4d2e-8f7b-4bea957c3da1_characterizeImage_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93787240-5f77-4d2e-8f7b-4bea957c3da1_characterizeImage_1039674_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:50 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:50 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10eda5fd34e16b8cf019160f88e52a48 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 2fb60b54aa7a6fb3f5787460ee725295 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/58ba113c-8281-4c58-a899-fe66658994ec_calibrate_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/58ba113c-8281-4c58-a899-fe66658994ec_calibrate_1039672_27.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6321dd04-3729-4750-ad17-ac2b34dc0ec9_characterizeImage_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6321dd04-3729-4750-ad17-ac2b34dc0ec9_characterizeImage_1039672_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:51 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bbad86bb971fcd66f0d678aae5f1406 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash c617e20cdc607be9f73f88b4bb6945cd parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8afec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2f347c0a-d2e0-4b2c-9823-eeaa2729692e_calibrate_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2f347c0a-d2e0-4b2c-9823-eeaa2729692e_calibrate_1039674_37.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/eff8265a-bbb0-4ac1-9602-314ad50fb2e1_characterizeImage_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/eff8265a-bbb0-4ac1-9602-314ad50fb2e1_characterizeImage_1039674_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:51 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 70 try 1 failed parsl.dataflow.dflow ERROR: Task 70 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4808d5e5f8ae0182bed8918210634ad8 with result from task 70 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 71 try 0 failed parsl.dataflow.dflow INFO: Task 71 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d921e44d-785c-49d0-9a5a-15957edd1fbb_calibrate_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d921e44d-785c-49d0-9a5a-15957edd1fbb_calibrate_1039768_2.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f63ad9ea-e428-4e44-bb79-4b78145fd7bf_characterizeImage_1039768_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 34 try 1 failed parsl.dataflow.dflow ERROR: Task 34 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry d1ffef71e3498f373260a37e9f28ddb0 with result from task 34 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 35 try 0 failed parsl.dataflow.dflow INFO: Task 35 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/4743e91d-ca4e-46ba-8da9-96d9b0ab7417_calibrate_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/4743e91d-ca4e-46ba-8da9-96d9b0ab7417_calibrate_1039673_2.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6950df1d-b235-4c0b-93a3-3a5720101dfd_characterizeImage_1039673_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:52 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5865b67eb47de0b7ab83b93ad12724b with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash dc11a859a808e53b2c32ef0b65e824fe parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/199981ba-e442-4994-b11b-1acf5c5d7cf5_calibrate_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/199981ba-e442-4994-b11b-1acf5c5d7cf5_calibrate_1039673_7.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5765792d-4e84-44f2-beb3-3e97f20cc54d_characterizeImage_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5765792d-4e84-44f2-beb3-3e97f20cc54d_characterizeImage_1039673_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:52 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:53 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:53 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:54 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:54 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62e21b7b18d6b60b201116552a063a76 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 8b1d8d51567186b30d96bb93164d5144 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/38cc02dd-7066-4fb1-b7bd-13cbca40f02a_calibrate_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/38cc02dd-7066-4fb1-b7bd-13cbca40f02a_calibrate_1039673_18.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/e4305031-2f3f-4162-b91e-fbdc1a461c09_characterizeImage_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/e4305031-2f3f-4162-b91e-fbdc1a461c09_characterizeImage_1039673_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:55 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55a512d336e5f144e092de75cf2f2a8c with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash ee5e8ea2c0e273b33c6034a5497a38c0 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/271c0ae4-0dfc-450d-bf4c-af0d08bffd2e_calibrate_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/271c0ae4-0dfc-450d-bf4c-af0d08bffd2e_calibrate_1039672_42.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a0eba6d8-10f4-4672-948d-9ac45821e746_characterizeImage_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a0eba6d8-10f4-4672-948d-9ac45821e746_characterizeImage_1039672_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:55 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:55 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:56 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:57 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 859 try 0 failed parsl.dataflow.dflow INFO: Task 859 marked for retry parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 7799623fcd581f14738a27bd318423f3 parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 1 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db11ee834e25bc47c8ac80d0fa5ad379 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 33eb4bd331b14d0ef818fb60764157e6 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f912c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/414e30f3-eccd-4adb-933f-de2de2619e0d_calibrate_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/414e30f3-eccd-4adb-933f-de2de2619e0d_calibrate_1039674_36.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/26ee19f5-d188-4385-a275-5cc0c6558369_characterizeImage_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/26ee19f5-d188-4385-a275-5cc0c6558369_characterizeImage_1039674_36.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eec5ed102f9f1c0d35c22c88d1c446a with result from task 1102 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:57 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash b52602cfce99969a2299535637954f00 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/11816491-dcca-4207-8859-21b119265be2_calibrate_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/11816491-dcca-4207-8859-21b119265be2_calibrate_1039673_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/dca5480b-6f8d-4231-bcf9-236d81d146f5_characterizeImage_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/dca5480b-6f8d-4231-bcf9-236d81d146f5_characterizeImage_1039673_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:58 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:58 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:59 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fcde757426a640d131faff0fb648127 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash d532bdfa6e0202aa6237e207a6d4ba4d parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ec67b3c1-c5ba-4824-96aa-1078619fe4b9_calibrate_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ec67b3c1-c5ba-4824-96aa-1078619fe4b9_calibrate_1039672_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5b2b4ae-5298-45c5-8396-96c9c762ac44_characterizeImage_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5b2b4ae-5298-45c5-8396-96c9c762ac44_characterizeImage_1039672_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:39:59 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:00 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:00 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:01 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b2dc2aeb056a91953a6d5d7262e7d45 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 5ae04ad9c622ceeb691d12727bde2f17 parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/1e9b79b1-b1fa-438b-8371-9e6d02ecc787_calibrate_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/1e9b79b1-b1fa-438b-8371-9e6d02ecc787_calibrate_1039674_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/512664e1-6b53-474a-8719-374c16b7e5f2_characterizeImage_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/512664e1-6b53-474a-8719-374c16b7e5f2_characterizeImage_1039674_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:02 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df680e162b448748e0c5577671df6012 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash 6282aea18664149395702b11b8a670c9 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/0e9a2f00-1ba8-4c24-8b30-ef9539eceaa2_calibrate_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/0e9a2f00-1ba8-4c24-8b30-ef9539eceaa2_calibrate_1039673_50.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9a3aa1c6-f5bc-4f29-bfc6-ac799b72320b_characterizeImage_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9a3aa1c6-f5bc-4f29-bfc6-ac799b72320b_characterizeImage_1039673_50.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 745 try 0 failed parsl.dataflow.dflow INFO: Task 745 marked for retry parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash 0434ae480211595d5fa2a12c13a21f93 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 1 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:02 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:03 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb3d97b79721b6d83b47a45f77e7cd64 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash fe61226bb3c90eeed680c42be0ee2edb parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/230117a6-7579-4819-b790-cb043c7e3774_calibrate_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/230117a6-7579-4819-b790-cb043c7e3774_calibrate_1039672_40.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4dafcc86-0c02-470c-b79a-c31aa58cf463_characterizeImage_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4dafcc86-0c02-470c-b79a-c31aa58cf463_characterizeImage_1039672_40.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 968b9381372f66dfc80d0b1970fa9629 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 2c9472c9ec54ef3ada4897ac7e3a533c parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/aaf26a78-4fde-403e-b667-0aa592e6634f_calibrate_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/aaf26a78-4fde-403e-b667-0aa592e6634f_calibrate_1039673_27.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3435543c-793c-4c7f-adaa-03d4d0ca2690_characterizeImage_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3435543c-793c-4c7f-adaa-03d4d0ca2690_characterizeImage_1039673_27.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7f0153b0c0e35a26658de17d21d8d49 with result from task 1078 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:04 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash 6f92f21de8b86940e90ebea1d6c0f4a2 parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/bc6aa2f7-dd30-4180-8ef8-06d0a047b580_calibrate_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/bc6aa2f7-dd30-4180-8ef8-06d0a047b580_calibrate_1039673_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/94c7ab4b-6f6a-43d1-a27d-e207a5111230_characterizeImage_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/94c7ab4b-6f6a-43d1-a27d-e207a5111230_characterizeImage_1039673_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:04 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 470b4da64521f297b97f445d215e2895 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash ac115b651f636e4c77b5bbc2c242f35c parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/56d3ace8-1d4e-4211-aa89-841eb1b3dc70_calibrate_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/56d3ace8-1d4e-4211-aa89-841eb1b3dc70_calibrate_1039672_50.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/32c7d364-e609-49e3-83aa-6acdcf102579_characterizeImage_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/32c7d364-e609-49e3-83aa-6acdcf102579_characterizeImage_1039672_50.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55301090325756c7e7a35e93c7a4047b with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash b9a5b7fa57eebcbef4d0bc75658a6641 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fd480e72-8a81-4456-814e-4640b5852389_calibrate_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fd480e72-8a81-4456-814e-4640b5852389_calibrate_1039768_43.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/093fd1c6-6746-4170-bc5e-5ffad1404048_characterizeImage_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/093fd1c6-6746-4170-bc5e-5ffad1404048_characterizeImage_1039768_43.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 991 try 0 failed parsl.dataflow.dflow INFO: Task 991 marked for retry parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 6fb14b01f918eda680180fe48e1156a4 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 1 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 987b495a410b95b04335e84669d967e2 with result from task 1090 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:04 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 089bfcd56525e9242f1343e0172da518 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e217ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/75d93b83-31d6-42d5-8d3c-ac0378db0874_calibrate_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/75d93b83-31d6-42d5-8d3c-ac0378db0874_calibrate_1039674_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/de9b8ff7-556c-4f78-bb3c-33326ad64e65_characterizeImage_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/de9b8ff7-556c-4f78-bb3c-33326ad64e65_characterizeImage_1039674_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:05 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:05 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b8691f23694d78e4c798e5801734b7 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash cbcad1e2b1002dbd3b10c76eee33ee29 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8afc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/41b196af-4aed-4828-9ea9-86360adb9724_calibrate_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/41b196af-4aed-4828-9ea9-86360adb9724_calibrate_1039673_25.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c75243c2-ecf9-4dce-9b40-8cc74d2abaed_characterizeImage_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c75243c2-ecf9-4dce-9b40-8cc74d2abaed_characterizeImage_1039673_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:06 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:06 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:07 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:08 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd3be456fa2e776196be2a9fa36ae650 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash fd8273b84c60426cdac2d67a8db9f75f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/e00a0c82-fa76-4e82-96be-093bfbb912b4_calibrate_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/e00a0c82-fa76-4e82-96be-093bfbb912b4_calibrate_1039770_21.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/3a75ba85-39ce-4f18-ae7f-714f163c973f_characterizeImage_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/3a75ba85-39ce-4f18-ae7f-714f163c973f_characterizeImage_1039770_21.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c13f72f8baeda394d33fd05082ee6269 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash b103cce0fc91448102160347fbdb8125 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5056c067-f576-481b-8281-c3fc07618607_calibrate_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5056c067-f576-481b-8281-c3fc07618607_calibrate_1039673_48.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7693f361-b390-41a0-a43d-fdb3beab453d_characterizeImage_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7693f361-b390-41a0-a43d-fdb3beab453d_characterizeImage_1039673_48.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ccd8edfc6f2af0d3bbb638859c4a105 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash a909d501a4a62c1d93e30f16dc08e081 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8adc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/08c9f3e1-7cc5-4a31-9ed0-e64635502a70_calibrate_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/08c9f3e1-7cc5-4a31-9ed0-e64635502a70_calibrate_1039674_23.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6a98c9f3-d213-494a-aa1d-b623561b8e6a_characterizeImage_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6a98c9f3-d213-494a-aa1d-b623561b8e6a_characterizeImage_1039674_23.stderr parsl.dataflow.dflow INFO: Task 1105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 100878fa02f63f4dd175ec8feed28fa4 with result from task 1105 parsl.dataflow.dflow DEBUG: Task 1105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1106 has memoization hash 48d1a077f5673fe2bb58ebac2b075bda parsl.dataflow.memoization INFO: Task 1106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1106 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1d1fade6-2168-4b8c-bf01-5472027eba2b_calibrate_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1d1fade6-2168-4b8c-bf01-5472027eba2b_calibrate_1039672_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1328c4b9-8025-4a97-bb05-202a006c45c5_characterizeImage_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1328c4b9-8025-4a97-bb05-202a006c45c5_characterizeImage_1039672_51.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e7bbe3174a85ea919212a0b7bf3e22 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash bc2a6615dfd8e4aa6297a8ed6fd2eaa7 parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3872e470-b259-4bc6-8490-94bd8a255f59_calibrate_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3872e470-b259-4bc6-8490-94bd8a255f59_calibrate_1039770_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cf228451-63ac-42ee-900d-d1ec0b3717a6_characterizeImage_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cf228451-63ac-42ee-900d-d1ec0b3717a6_characterizeImage_1039770_17.stderr parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 715ee1f2861304f696ba37067208c2fd with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash a80fa4be791584adc2e3f444dff42ca1 parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2179c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b8cb098f-d730-406b-b0a6-bb6f51d197db_calibrate_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b8cb098f-d730-406b-b0a6-bb6f51d197db_calibrate_1039673_39.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/197f2221-9f27-42a5-999d-f7732dce25c7_characterizeImage_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/197f2221-9f27-42a5-999d-f7732dce25c7_characterizeImage_1039673_39.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f5afdc194dd7d501775803908285256 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 9d085c3749f78105b3dbee0511776542 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/89628530-281d-4c7c-b7c6-bd99eebe41ef_calibrate_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/89628530-281d-4c7c-b7c6-bd99eebe41ef_calibrate_1039673_22.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1b1306-f27a-4617-a12a-1e25f9a575a0_characterizeImage_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1b1306-f27a-4617-a12a-1e25f9a575a0_characterizeImage_1039673_22.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34efcbbced1d463f7ef6b1d5dafcd96a with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash f614f8e3f40bb4f5d472213410001da8 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6c3fd182-e139-4cfd-9725-ef75181de375_calibrate_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6c3fd182-e139-4cfd-9725-ef75181de375_calibrate_1039768_32.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/44673071-e92a-409d-96cc-7481821f1efb_characterizeImage_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/44673071-e92a-409d-96cc-7481821f1efb_characterizeImage_1039768_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:09 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51bfdde2367509a3ad00d8591bd45a1a with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash cca485f1729694a67bf511d3338637ad parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/113e5afe-9857-4b3b-9c1f-c5109921e414_calibrate_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/113e5afe-9857-4b3b-9c1f-c5109921e414_calibrate_1039672_52.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4213b436-7abf-43bd-bf9e-216d5695015e_characterizeImage_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/4213b436-7abf-43bd-bf9e-216d5695015e_characterizeImage_1039672_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:09 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2603df5a9fb03d781d9512f36f054bd4 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 4ce3a929d38b045d78db6057719dfa37 parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0754b61d-dcd0-45fc-9b4c-7ead0c64659d_calibrate_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0754b61d-dcd0-45fc-9b4c-7ead0c64659d_calibrate_1039674_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8c9bb852-e3db-49d4-8e2b-48649d79968a_characterizeImage_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8c9bb852-e3db-49d4-8e2b-48649d79968a_characterizeImage_1039674_38.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad524bd5e01baf02f36487dabbfe36f with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 543dae1fc2130a44a6ad018121169742 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8306abd2-ad7a-4a49-9543-e8ab20a7c50d_calibrate_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8306abd2-ad7a-4a49-9543-e8ab20a7c50d_calibrate_1039673_36.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/0fff7db2-dd27-48d5-a5b5-380208a6c8c2_characterizeImage_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/0fff7db2-dd27-48d5-a5b5-380208a6c8c2_characterizeImage_1039673_36.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e911868c1b0b8641aaad68a1e0d3ab4 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 8b2dccf709a8e2b88f37d57029a21045 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ccba0100-9ff6-4a84-9387-f4f3ce39fd57_calibrate_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ccba0100-9ff6-4a84-9387-f4f3ce39fd57_calibrate_1039770_29.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/f85fbb4a-f03d-4a83-b664-a7abb06661cd_characterizeImage_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/f85fbb4a-f03d-4a83-b664-a7abb06661cd_characterizeImage_1039770_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:10 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c24eb0c477817115b3a28e0490adf3e with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 75fa3a4ae7eaa16bf9d71267db8c2f5b parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6c582ab1-5008-42ba-aaaf-1c7333ee41bc_calibrate_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6c582ab1-5008-42ba-aaaf-1c7333ee41bc_calibrate_1039672_15.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fa9b59ba-625d-4507-90af-4452ee1a0dc9_characterizeImage_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fa9b59ba-625d-4507-90af-4452ee1a0dc9_characterizeImage_1039672_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:10 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3d11955308e52b7aba778c6660b2bac with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash d4635b447660e22ab0b6357cb579a9bd parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/924a0756-0b25-4e91-bb54-b5c5aa526ece_calibrate_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/924a0756-0b25-4e91-bb54-b5c5aa526ece_calibrate_1039674_21.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/d32e2410-a74c-4808-8c58-72af9318f3cb_characterizeImage_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/d32e2410-a74c-4808-8c58-72af9318f3cb_characterizeImage_1039674_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:11 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:11 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce29401fdbd13439aa89e40b7aa123b1 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 05a13fa3ff703f17b81cb422dd74bbfe parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9707bcd3-add0-4450-85d1-0c165ab852d8_calibrate_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9707bcd3-add0-4450-85d1-0c165ab852d8_calibrate_1039674_4.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/971d3828-ee6e-460d-bc6f-465afbdb30fa_characterizeImage_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/971d3828-ee6e-460d-bc6f-465afbdb30fa_characterizeImage_1039674_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:12 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70ed9c5ae6c1e0eb316832788aa5ffc with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 2596b6ac74172793272a39db190eca26 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7888171e-db21-481a-bda0-fd8bdd4bddda_calibrate_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7888171e-db21-481a-bda0-fd8bdd4bddda_calibrate_1039672_46.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fb85db6-b53f-4852-be7e-16daf753d118_characterizeImage_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fb85db6-b53f-4852-be7e-16daf753d118_characterizeImage_1039672_46.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9201217981896a046e39caf81eeff8 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 35fa94f31c3886d866bd348c6df698ee parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e7889509-b738-4b6c-95e3-a87476f51637_calibrate_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e7889509-b738-4b6c-95e3-a87476f51637_calibrate_1039769_28.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/6812b309-2544-4edd-8d27-89b212b280ab_characterizeImage_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/6812b309-2544-4edd-8d27-89b212b280ab_characterizeImage_1039769_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:12 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:13 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37ef0ec04d459dc3fbe71834452f562f with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 240fc457750c072a6de124e799eb7319 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8acae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/e8dfc5a1-6586-4d4a-9a10-ea0ae89b431a_calibrate_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/e8dfc5a1-6586-4d4a-9a10-ea0ae89b431a_calibrate_1039672_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1092e414-cfa2-45cb-9ee0-b3eb99d12407_characterizeImage_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1092e414-cfa2-45cb-9ee0-b3eb99d12407_characterizeImage_1039672_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 369 parsl.dataflow.strategy DEBUG: Executor multi has 369 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:40:13 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:40:13 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:14 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea229f37cfd66cba6f21e4e4bef4724 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 4c1054f132b59b2b8fb9ec9cc8a0d508 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c1939e4b-6909-4d01-b881-7fa1d6edb8e8_calibrate_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c1939e4b-6909-4d01-b881-7fa1d6edb8e8_calibrate_1039674_60.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c60f5355-119c-43e1-b577-46e6a179336f_characterizeImage_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c60f5355-119c-43e1-b577-46e6a179336f_characterizeImage_1039674_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:14 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 745 try 1 failed parsl.dataflow.dflow ERROR: Task 745 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0434ae480211595d5fa2a12c13a21f93 with result from task 745 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 746 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 746 try 0 failed parsl.dataflow.dflow INFO: Task 746 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/32272889-1408-473a-b589-81407ebec974_calibrate_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/32272889-1408-473a-b589-81407ebec974_calibrate_1039770_2.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/02964067-1449-43d8-b8ff-6d859e6c8897_characterizeImage_1039770_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:15 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 859 try 1 failed parsl.dataflow.dflow ERROR: Task 859 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 7799623fcd581f14738a27bd318423f3 with result from task 859 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 860 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 860 try 0 failed parsl.dataflow.dflow INFO: Task 860 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/089598ea-05c1-437d-b675-2dfd568d7d87_calibrate_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/089598ea-05c1-437d-b675-2dfd568d7d87_calibrate_1039674_2.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2170640e-e2dd-450a-ab5d-f5bb4672e24b_characterizeImage_1039674_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:15 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:15 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:16 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd31142655d74b578e2fc77361c6398e with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 85ba3bedf4104245db52fd58b1ddacc9 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1436f68e-9d4c-44b8-b255-1c34210d554b_calibrate_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1436f68e-9d4c-44b8-b255-1c34210d554b_calibrate_1039770_10.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/868aa207-d264-4ba2-b7e3-528e0d401fda_characterizeImage_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/868aa207-d264-4ba2-b7e3-528e0d401fda_characterizeImage_1039770_10.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37a881a5152f1e6eb48db27247533c34 with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash bc5a38dd84b175ca799cd62a8e98887e parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ed00b233-18bb-44f9-975c-28110ad0272c_calibrate_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ed00b233-18bb-44f9-975c-28110ad0272c_calibrate_1039674_8.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/dbba8623-79cc-45cd-87dc-c539037c9d2d_characterizeImage_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/dbba8623-79cc-45cd-87dc-c539037c9d2d_characterizeImage_1039674_8.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 991 try 1 failed parsl.dataflow.dflow ERROR: Task 991 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb14b01f918eda680180fe48e1156a4 with result from task 991 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 992 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 992 try 0 failed parsl.dataflow.dflow INFO: Task 992 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/8c91c037-d829-425d-8561-83009a65f281_calibrate_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/8c91c037-d829-425d-8561-83009a65f281_calibrate_1039769_2.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a8d14e-39af-417a-a2e9-8b23394fd3cc_characterizeImage_1039769_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:16 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cea29e4b9f05c1d6e38427df2d263d8 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash fe02373fba432b7a3bf1c9979e615335 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f2c583bc-78b4-4787-baec-588282914a42_calibrate_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f2c583bc-78b4-4787-baec-588282914a42_calibrate_1039672_56.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/258a6030-a68f-4c16-9e69-1e33ca2ae532_characterizeImage_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/258a6030-a68f-4c16-9e69-1e33ca2ae532_characterizeImage_1039672_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:17 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a2fa50a69d1fe80374a884dcc4b6c2d with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash cfa764272a90a443094417bea87acad5 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/94c7da42-339a-4fb1-9147-a6739ab75526_calibrate_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/94c7da42-339a-4fb1-9147-a6739ab75526_calibrate_1039674_48.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8da4a513-2333-4b16-b821-0900119cfa13_characterizeImage_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/8da4a513-2333-4b16-b821-0900119cfa13_characterizeImage_1039674_48.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0de2bd72f1f4106cfdf2dc4e41fe0494 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 66f549662ed00b31c73124982a1c337c parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ecb549db-3709-460a-a3db-049dc18a5369_calibrate_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ecb549db-3709-460a-a3db-049dc18a5369_calibrate_1039768_55.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4e0fdea0-6fa0-42db-8d23-d9bcc569384a_characterizeImage_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4e0fdea0-6fa0-42db-8d23-d9bcc569384a_characterizeImage_1039768_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:17 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd7fff069a399ccf118b16881342b2a7 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 31be7e0d18c092297d2db84f087589d2 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/610780cd-0a37-4993-a387-8b0574f0f671_calibrate_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/610780cd-0a37-4993-a387-8b0574f0f671_calibrate_1039673_21.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/078297e9-914b-42b8-a55c-79908846fe02_characterizeImage_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/078297e9-914b-42b8-a55c-79908846fe02_characterizeImage_1039673_21.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abdc1b5f540abf0c46fc334f817deaf4 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 06af83fe06c655c18d6fbd7178ffa3fc parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8acea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/63baf1b0-066d-4771-be40-140c26eb8ded_calibrate_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/63baf1b0-066d-4771-be40-140c26eb8ded_calibrate_1039768_31.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d9c40a07-c3d8-438b-b058-eec38eb68f1b_characterizeImage_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d9c40a07-c3d8-438b-b058-eec38eb68f1b_characterizeImage_1039768_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:18 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45a39ee549df54d7d2ad20e8ed31a775 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 9dd951d27e2ed218605b5d031df3d424 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/95bc0b02-0dd8-4148-b199-69dfd292720b_calibrate_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/95bc0b02-0dd8-4148-b199-69dfd292720b_calibrate_1039770_36.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1c7568aa-efb7-4dab-846e-7ca56a69bbb3_characterizeImage_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1c7568aa-efb7-4dab-846e-7ca56a69bbb3_characterizeImage_1039770_36.stderr parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07bfbbb997e110b35b37ddae099dabc8 with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash f4891e7773050a3e31dbd7eceebeaf44 parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0742c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4699c771-a670-46c8-acfd-797fe80208bd_calibrate_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4699c771-a670-46c8-acfd-797fe80208bd_calibrate_1039770_18.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/48c4d3ae-e724-4b12-a63f-75a9ad973c6e_characterizeImage_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/48c4d3ae-e724-4b12-a63f-75a9ad973c6e_characterizeImage_1039770_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:18 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16e5d60e6958d0e1a5d67f65d8f5d76b with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 6bd1d7ae9cf184f629a65c185bd20049 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8af380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d6c78836-12c9-497c-8307-6bcfc13779cb_calibrate_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d6c78836-12c9-497c-8307-6bcfc13779cb_calibrate_1039770_20.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/710c25d2-fc89-4ebb-b50e-fea4eb21713d_characterizeImage_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/710c25d2-fc89-4ebb-b50e-fea4eb21713d_characterizeImage_1039770_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:19 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8390a8ffd32022ce99f5eccd03136c0 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash a4b6f4ad6408e257cd6904dada9f611b parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/df915a90-ce60-46b0-a0aa-32ecbf3473b2_calibrate_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/df915a90-ce60-46b0-a0aa-32ecbf3473b2_calibrate_1039768_46.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f80c6eca-0fed-470f-a287-dcc28e88a634_characterizeImage_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f80c6eca-0fed-470f-a287-dcc28e88a634_characterizeImage_1039768_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:19 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855b723644171f6109bea49da4ebbbb6 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash bf99c7630809c122968434f7af3aa388 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3ea7a349-dcbf-47dc-b15f-7e08f4d7f1ad_calibrate_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3ea7a349-dcbf-47dc-b15f-7e08f4d7f1ad_calibrate_1039769_46.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f812cbce-8e10-4b56-934b-5e46296e53c5_characterizeImage_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f812cbce-8e10-4b56-934b-5e46296e53c5_characterizeImage_1039769_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:19 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf6807fa6473ab24ef678a83236d8746 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 2c69238b93bb10415ab85be27b164f8b parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2176a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b7b4a2bb-5e83-49ed-a684-4faefb869531_calibrate_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b7b4a2bb-5e83-49ed-a684-4faefb869531_calibrate_1039770_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/27202d2c-a53b-47e9-a409-133f05cd4d2a_characterizeImage_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/27202d2c-a53b-47e9-a409-133f05cd4d2a_characterizeImage_1039770_22.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359f990620a15874d1a3c79635d427d6 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash abd8155bca964e2af15dfb6c418fe9fe parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e0263606-0d58-4325-98e4-e548ccc85456_calibrate_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e0263606-0d58-4325-98e4-e548ccc85456_calibrate_1039673_17.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/f65a691b-ddf5-439c-b8a0-49d9ed910e09_characterizeImage_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/f65a691b-ddf5-439c-b8a0-49d9ed910e09_characterizeImage_1039673_17.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c76faa5d3a0e67fbc1fac0233655d27f with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 2d17d8fcfb3163267d007bc502cde381 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2796c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0b2ae7ec-0ac6-4c56-be2d-5b6c9f3b277c_calibrate_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0b2ae7ec-0ac6-4c56-be2d-5b6c9f3b277c_calibrate_1039674_28.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c42385a1-28e7-47e7-881b-f1db37647fde_characterizeImage_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/c42385a1-28e7-47e7-881b-f1db37647fde_characterizeImage_1039674_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:21 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce401e43a1970a812fdb048fbdd16df with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 9ccf724c0f15cf4754b62ba342c84a5c parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/312a9cf8-b993-4a6d-9dc5-5a6a02233509_calibrate_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/312a9cf8-b993-4a6d-9dc5-5a6a02233509_calibrate_1039674_49.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4d3795fa-de41-4aa3-b1db-1c0982a305cd_characterizeImage_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4d3795fa-de41-4aa3-b1db-1c0982a305cd_characterizeImage_1039674_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:22 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:22 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 816d12c01bb7d9de22c13b399af9a91a with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 5333ed3f908e7e69353584e19697ebeb parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1ee9b0e2-dfde-4513-9e87-0b9cfc3998fc_calibrate_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1ee9b0e2-dfde-4513-9e87-0b9cfc3998fc_calibrate_1039770_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/37908053-dba9-4815-a624-c31c3c8cd487_characterizeImage_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/37908053-dba9-4815-a624-c31c3c8cd487_characterizeImage_1039770_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:22 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afc66dc8d4e4640e097a6bd08728dcbc with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 870a48b7e79605eed7235b903a276c34 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0706035f-9cec-459b-b598-1ee8165a283b_calibrate_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0706035f-9cec-459b-b598-1ee8165a283b_calibrate_1039672_13.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/81ad6a0a-6746-49ed-b0f5-6cb5d47319fc_characterizeImage_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/81ad6a0a-6746-49ed-b0f5-6cb5d47319fc_characterizeImage_1039672_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:23 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:23 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 773e794af1ac46a51071594b5f4a1817 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 4e51054587aff0887b01caf58bb60aed parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b43f29bc-77f7-44bb-9380-29f92861e589_calibrate_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b43f29bc-77f7-44bb-9380-29f92861e589_calibrate_1039673_49.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9d835180-73fb-4e6c-b680-5070339b403a_characterizeImage_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9d835180-73fb-4e6c-b680-5070339b403a_characterizeImage_1039673_49.stderr parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52b4810f94b01ee8125cb1d81203898d with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash ee842016914c67cfd88d5683bd4dc85f parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3348d4a-f326-484a-ad13-e5e3c5289d24_calibrate_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3348d4a-f326-484a-ad13-e5e3c5289d24_calibrate_1039673_29.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c6cf1c7d-70b7-43cf-84d6-947b174f76ac_characterizeImage_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c6cf1c7d-70b7-43cf-84d6-947b174f76ac_characterizeImage_1039673_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:23 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 182dbd14ac29a617f66f07f37843bfe6 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 8ffd4f22abf60ebf10ad3e9ebc8f11e1 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0357ec04-a2b6-4d5b-bb7d-81fc7d84f2a3_calibrate_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0357ec04-a2b6-4d5b-bb7d-81fc7d84f2a3_calibrate_1039769_14.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/97efbac6-69e9-410e-8cf9-fc72389ae604_characterizeImage_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/97efbac6-69e9-410e-8cf9-fc72389ae604_characterizeImage_1039769_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:24 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9318c5445167ff0f489b9a6f478efd29 with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash 537a3dc37841327cb27963f051a2ad81 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6cad7ec2-fcaf-4caa-bfbe-fe8a43c356d0_calibrate_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6cad7ec2-fcaf-4caa-bfbe-fe8a43c356d0_calibrate_1039674_24.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4eebf4fc-cfe5-49c4-8b9d-010f33326ff9_characterizeImage_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4eebf4fc-cfe5-49c4-8b9d-010f33326ff9_characterizeImage_1039674_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:24 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:25 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9c53f8a63a46903ffa3b9afcbe319a with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 349df24d3ae3b408a6df318dfbe53447 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6a2bdf4e-09b2-4532-847b-3a3277246cc6_calibrate_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6a2bdf4e-09b2-4532-847b-3a3277246cc6_calibrate_1039674_12.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/00fabb0a-c111-43e9-a08d-688816824f51_characterizeImage_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/00fabb0a-c111-43e9-a08d-688816824f51_characterizeImage_1039674_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:25 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:25 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:26 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:26 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:27 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae23b3f1043a4182d26a9ef301f40f3e with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash ecd368078db68ac49f5dfbbc6443a185 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ca75a4d8-d78b-4298-b756-028e4d383e5d_calibrate_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ca75a4d8-d78b-4298-b756-028e4d383e5d_calibrate_1039768_50.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/1099bcda-4c7a-4f05-8023-36e937398db3_characterizeImage_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/1099bcda-4c7a-4f05-8023-36e937398db3_characterizeImage_1039768_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:27 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b7e599a170b496970693f6bda3ba2ee with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash a068ace008da661cc8622162c2d6001e parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1ddeb693-f6d4-4038-8f05-e649341a940f_calibrate_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1ddeb693-f6d4-4038-8f05-e649341a940f_calibrate_1039672_35.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6933caf3-f0c9-47b2-a35c-83c7b22e4a43_characterizeImage_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/6933caf3-f0c9-47b2-a35c-83c7b22e4a43_characterizeImage_1039672_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:28 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ca5c2ae5d2d0ee6f937acee12eb4dc with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 27685f719af289331f651b5e11bb4bae parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8504a2fe-97f1-4199-8b06-b3ca72afbda1_calibrate_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8504a2fe-97f1-4199-8b06-b3ca72afbda1_calibrate_1039674_30.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0b4acbd4-b418-4a27-94ae-4606e7830e3e_characterizeImage_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0b4acbd4-b418-4a27-94ae-4606e7830e3e_characterizeImage_1039674_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:28 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70b4ff97275aebd0a837709f93705c22 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 14ef0666ed3690a08eed2431758c5e19 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4ce6af6c-f6f2-4cd9-99ea-4c060ccf3a18_calibrate_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4ce6af6c-f6f2-4cd9-99ea-4c060ccf3a18_calibrate_1039674_1.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a6a98c36-9bcd-4a5c-905c-98049b8b3dfd_characterizeImage_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a6a98c36-9bcd-4a5c-905c-98049b8b3dfd_characterizeImage_1039674_1.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17a468ec19383d49f8fd0bd63a3385bb with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 246ac5c581fb5d36c8e4265dafc39718 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5c0f129d-8277-4af3-87fa-fedd0f7da289_calibrate_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5c0f129d-8277-4af3-87fa-fedd0f7da289_calibrate_1039768_53.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/097c9347-50d8-45e1-9f00-d8fcb2dfd968_characterizeImage_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/097c9347-50d8-45e1-9f00-d8fcb2dfd968_characterizeImage_1039768_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:29 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15ce1ba5a16f40a918c5a8172d5949ff with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 3bf181174cd926ca98e04ea0af19c111 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/7b9cc0a9-13a1-42d5-9b7e-bd8cb86518e0_calibrate_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/7b9cc0a9-13a1-42d5-9b7e-bd8cb86518e0_calibrate_1039768_16.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/732e6f15-f442-4ba2-aec9-faf666558a36_characterizeImage_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/732e6f15-f442-4ba2-aec9-faf666558a36_characterizeImage_1039768_16.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb83e747714d000469f692f5c9dce35 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash f4b9340ee4012e25717d43a9e7cc0eb9 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2c1d0059-23ff-4f10-bc3f-430ef94e9182_calibrate_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2c1d0059-23ff-4f10-bc3f-430ef94e9182_calibrate_1039770_30.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/fe5ce469-9429-473d-b1c6-0933bb3f1b1d_characterizeImage_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/fe5ce469-9429-473d-b1c6-0933bb3f1b1d_characterizeImage_1039770_30.stderr parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab0ff9d9d8af1da7424bdcef98bbdc7 with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash 78934d0633227cea138003e52261001c parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0756c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/064ee654-a579-4698-90f2-a77fe924a150_calibrate_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/064ee654-a579-4698-90f2-a77fe924a150_calibrate_1039768_25.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8034ac87-5ede-423e-adc4-0196cb1bb030_characterizeImage_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8034ac87-5ede-423e-adc4-0196cb1bb030_characterizeImage_1039768_25.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5ca329cb46a76bceabd61c80cac9af9 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 5437543d2d9dc87bdff07f7100456a27 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ae7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14fe8bf9-30c5-4ca9-ad29-b5b2f4f5b622_calibrate_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14fe8bf9-30c5-4ca9-ad29-b5b2f4f5b622_calibrate_1039674_6.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/fb8f627f-8a6a-4b28-871a-69d03d20b186_characterizeImage_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/fb8f627f-8a6a-4b28-871a-69d03d20b186_characterizeImage_1039674_6.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493b4d8f27ae8d7d8178fed41bfa0c52 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash ffe139607cba30c2738a50919d2ad5cc parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5950737b-6473-4a2a-a39c-613dadcee13f_calibrate_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5950737b-6473-4a2a-a39c-613dadcee13f_calibrate_1039769_34.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/677f622b-cb3c-4b4c-99e6-0df3489639cb_characterizeImage_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/677f622b-cb3c-4b4c-99e6-0df3489639cb_characterizeImage_1039769_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:30 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b459818656df5d94200e00bb161d982 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 7df5b3d194f81495ccab8336a243c5f0 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8add00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0c47d7d1-2925-493a-a40f-f2736324dfad_calibrate_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0c47d7d1-2925-493a-a40f-f2736324dfad_calibrate_1039769_24.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0be4fa6e-563e-4887-bf17-ed9f520f014b_characterizeImage_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0be4fa6e-563e-4887-bf17-ed9f520f014b_characterizeImage_1039769_24.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d66f95d879168e045612cd095d8480cc with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash b1cd223b7b43992a4e350b55be440fe6 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/c49847c2-ea14-4dc6-887a-ad301707365b_calibrate_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/c49847c2-ea14-4dc6-887a-ad301707365b_calibrate_1039768_3.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/68bd8e22-1d72-4b75-bda9-05656a0ec910_characterizeImage_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/68bd8e22-1d72-4b75-bda9-05656a0ec910_characterizeImage_1039768_3.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f01de1bb9667bc29c65aa0b81aa2d6f with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 922d150efa60ccdf5b2217c2f192850d parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fa738dbc-51a8-4765-a0fa-dbe1fa803ab7_calibrate_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fa738dbc-51a8-4765-a0fa-dbe1fa803ab7_calibrate_1039672_7.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fdd26a6-f37b-4b4f-9da3-aee8cba3d6e2_characterizeImage_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1fdd26a6-f37b-4b4f-9da3-aee8cba3d6e2_characterizeImage_1039672_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:31 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:32 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1bbd85fe3237163684b74880242d0b with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash cdd06ec11ee1833248913c55a14d501b parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/caa27223-efab-404f-b33b-ceefbe28ec3d_calibrate_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/caa27223-efab-404f-b33b-ceefbe28ec3d_calibrate_1039770_32.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/15b7d610-1bc4-4139-9e71-f4e0ba063228_characterizeImage_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/15b7d610-1bc4-4139-9e71-f4e0ba063228_characterizeImage_1039770_32.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4f0573ebf6b2feee9d649dfdc012e5d with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 1850972153f1cc0d4ba5155a06d91fd6 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bc44f467-b28b-4b83-b817-901a21308c47_calibrate_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bc44f467-b28b-4b83-b817-901a21308c47_calibrate_1039768_51.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a15267d4-26e2-42aa-a1e8-4b7479915d97_characterizeImage_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a15267d4-26e2-42aa-a1e8-4b7479915d97_characterizeImage_1039768_51.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65e4896610348f173f05a01b9ddc33ab with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 017b84588bb47f3a4d0524b7ddaeb092 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3f0fedcc-2d98-4620-a587-6283119d3e37_calibrate_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3f0fedcc-2d98-4620-a587-6283119d3e37_calibrate_1039673_19.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/04755abd-ba41-4f8c-97a9-72f1ba1bcd4f_characterizeImage_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/04755abd-ba41-4f8c-97a9-72f1ba1bcd4f_characterizeImage_1039673_19.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d69ffd47cc575427aa3e221a138cce49 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash bd8653e2f44c3783ecdbf6520610d2d0 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/71b9c34f-4ca7-4485-975b-f7925947a650_calibrate_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/71b9c34f-4ca7-4485-975b-f7925947a650_calibrate_1039672_17.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/877fa3ce-1b77-4cfe-b3c7-1e461cf118b1_characterizeImage_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/877fa3ce-1b77-4cfe-b3c7-1e461cf118b1_characterizeImage_1039672_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:33 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:34 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52f416e0ba15f220c025f3cece47f710 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash cc025759fea03fd444676502159f844d parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/dc22b7c7-a23f-4975-ba78-df381be9cdc3_calibrate_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/dc22b7c7-a23f-4975-ba78-df381be9cdc3_calibrate_1039769_36.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a2143ae2-2d97-4f9f-80dc-1dbc08fed8ca_characterizeImage_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a2143ae2-2d97-4f9f-80dc-1dbc08fed8ca_characterizeImage_1039769_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:35 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fbfdd904d89c3885c934bd88859b3bd with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash dfb4d50c66f12e7c7d603e84d6a4a076 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f9f7e911-8a0b-402a-a3a2-8fe45139cda8_calibrate_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f9f7e911-8a0b-402a-a3a2-8fe45139cda8_calibrate_1039674_22.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f6eeff25-5c9a-42ec-b748-780ce48cdb8d_characterizeImage_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f6eeff25-5c9a-42ec-b748-780ce48cdb8d_characterizeImage_1039674_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:35 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 563b07ec010a4e758c02e2f730bef9e3 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash f5e969191a5fd0d60875abb133cacde7 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5ece5783-2e17-4bcd-9693-cca992d7a776_calibrate_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5ece5783-2e17-4bcd-9693-cca992d7a776_calibrate_1039770_33.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/71e01d29-3314-47d9-83a6-66d8b10a8cec_characterizeImage_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/71e01d29-3314-47d9-83a6-66d8b10a8cec_characterizeImage_1039770_33.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58155b3385cb6eb75a3a588bb1574178 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 862e07e53b6c1714f4837a50bced6dfe parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dbefda3a-d4bd-4563-8e40-c25d729b44f1_calibrate_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dbefda3a-d4bd-4563-8e40-c25d729b44f1_calibrate_1039673_35.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3606cae8-38ad-4f71-91b2-713a42f8204d_characterizeImage_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3606cae8-38ad-4f71-91b2-713a42f8204d_characterizeImage_1039673_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:36 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c70556e45370727a098b69eccd25bb91 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 01c99cc0894e96564666b7ecdd693678 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/11cb396c-499d-4535-835a-794849576459_calibrate_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/11cb396c-499d-4535-835a-794849576459_calibrate_1039674_20.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4892641c-ae00-4f09-9493-826c8ae8f19b_characterizeImage_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4892641c-ae00-4f09-9493-826c8ae8f19b_characterizeImage_1039674_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:37 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dee6dda9e979f242d7e82c6136b728f with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 9ffa25e402fa29a495182d2fbe1ebea6 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9125c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3e01ffa-943d-4da6-a0da-ca1b3b32522f_calibrate_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3e01ffa-943d-4da6-a0da-ca1b3b32522f_calibrate_1039673_14.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f7dac3e-54b1-4071-ab4e-3c9dc52efdf8_characterizeImage_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3f7dac3e-54b1-4071-ab4e-3c9dc52efdf8_characterizeImage_1039673_14.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f93f177710c3e373b70af8b4b4769380 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 18aeb2da3ec0a6bd402ae11191eda86f parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8ca3c456-7bc9-4a79-a465-4a43a99421ef_calibrate_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8ca3c456-7bc9-4a79-a465-4a43a99421ef_calibrate_1039674_13.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4a922778-e8e7-4eb8-a275-b1820356a3be_characterizeImage_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/4a922778-e8e7-4eb8-a275-b1820356a3be_characterizeImage_1039674_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:37 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40324ee4746dc9625d0e4f5980058123 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 09c4dc14d3b646171fd41c2951f9cf81 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/882fb411-3ee0-486e-9ac9-ada124a309f1_calibrate_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/882fb411-3ee0-486e-9ac9-ada124a309f1_calibrate_1039770_41.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9c51f72f-3148-402e-952c-a7b4bf4bf7a1_characterizeImage_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9c51f72f-3148-402e-952c-a7b4bf4bf7a1_characterizeImage_1039770_41.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1efa82da371380349cce8250f27846d7 with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash ee819631b74025ae56e0ed71adf607fa parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f939b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c33c88cd-dced-4cd7-92b7-1ef6c1652bf2_calibrate_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c33c88cd-dced-4cd7-92b7-1ef6c1652bf2_calibrate_1039673_11.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bd105014-1948-4f50-8a6b-a2206178713f_characterizeImage_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bd105014-1948-4f50-8a6b-a2206178713f_characterizeImage_1039673_11.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd4c5657636372c8a95f5e52ee33a2a5 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 8855458028dd0d03bbdfa059cf6548ac parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2e019ceb-6bdb-4402-947e-5beaa39e787b_calibrate_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2e019ceb-6bdb-4402-947e-5beaa39e787b_calibrate_1039770_11.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2498c243-035e-4db5-9b53-c851205d774b_characterizeImage_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2498c243-035e-4db5-9b53-c851205d774b_characterizeImage_1039770_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:38 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db33117a60b0c6f5a082856a5ece9eaf with result from task 1099 parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 21ed6a644e9a28aa9d6df886ccbed6c9 parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ad152652-bc84-496c-ad09-facbe09383fa_calibrate_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ad152652-bc84-496c-ad09-facbe09383fa_calibrate_1039768_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8de2d7a6-a957-46c3-8026-5e45534a3e9b_characterizeImage_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8de2d7a6-a957-46c3-8026-5e45534a3e9b_characterizeImage_1039768_37.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74b828052bcef5a2ff4817e7fec8c3e7 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash cd58f851a044e976c9c91d1e68911406 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0156d1ca-ad77-488a-865d-17dbbebb6975_calibrate_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0156d1ca-ad77-488a-865d-17dbbebb6975_calibrate_1039672_16.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b11c25a4-0d72-4cf4-8b56-205a70dda6bc_characterizeImage_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b11c25a4-0d72-4cf4-8b56-205a70dda6bc_characterizeImage_1039672_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:38 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 489a7161106e94afb16c1dfa80c84b6f with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 4a3cba24a6b64e802eaf1810fbc79c6e parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f57ba267-e7a0-429d-a20a-604c80452296_calibrate_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f57ba267-e7a0-429d-a20a-604c80452296_calibrate_1039673_24.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7cd06872-25a9-4522-9efd-eb4334a893aa_characterizeImage_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7cd06872-25a9-4522-9efd-eb4334a893aa_characterizeImage_1039673_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:39 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe775f872fc9b62b988329ceec4f5cbc with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash b350c2607191667173b32bfd1d126c26 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f2f7ac68-ceb6-4227-8ef3-73800ddd8b72_calibrate_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f2f7ac68-ceb6-4227-8ef3-73800ddd8b72_calibrate_1039768_60.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/57ae3480-020f-474a-a8d5-b08aecf83014_characterizeImage_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/57ae3480-020f-474a-a8d5-b08aecf83014_characterizeImage_1039768_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:39 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:39 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:40 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27848a2d9a80dbb73bcc238e319efc16 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 710d38a0118a10acf954acb0a1c0cc23 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/72a74625-c8ec-4587-a327-c2e0d89f6d2d_calibrate_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/72a74625-c8ec-4587-a327-c2e0d89f6d2d_calibrate_1039770_52.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d5a3ec6f-a22b-466b-9be4-de88abc68992_characterizeImage_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d5a3ec6f-a22b-466b-9be4-de88abc68992_characterizeImage_1039770_52.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8f20d39be0b5359ee326c43218954c with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 0815193b285d9eabfefeb9de67c71f08 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9a809f4e-850a-4138-8e2a-d226b2810dda_calibrate_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9a809f4e-850a-4138-8e2a-d226b2810dda_calibrate_1039674_52.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2e3a89d1-fc94-4df4-8928-3af679da3e59_characterizeImage_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/2e3a89d1-fc94-4df4-8928-3af679da3e59_characterizeImage_1039674_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:40 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:41 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:41 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:41 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 510d1e4f21edea0299d2651cc9758359 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash aeba696421695b7880430989aca8db7f parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8ad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3fa36671-61a8-4848-a4af-1918ecb1aea2_calibrate_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3fa36671-61a8-4848-a4af-1918ecb1aea2_calibrate_1039768_14.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/6e881d82-d261-4f05-a890-e07ff6895109_characterizeImage_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/6e881d82-d261-4f05-a890-e07ff6895109_characterizeImage_1039768_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:42 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb305210ec2c1ad0faeaf282ef9d8769 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 44414b64146efb44358343e50c11b715 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f913880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/72052f47-0277-4a19-8da6-ccd5166dad50_calibrate_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/72052f47-0277-4a19-8da6-ccd5166dad50_calibrate_1039769_43.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3a416cc-6fb9-4f90-9af6-824a4c1387e9_characterizeImage_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3a416cc-6fb9-4f90-9af6-824a4c1387e9_characterizeImage_1039769_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:42 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:43 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:44 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:44 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16e724587f18841d031d3c411c21440 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash ce00c03ed0555ab4758d6a8769656124 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6e4fb53b-539d-4e00-a874-dc8ac39f19f2_calibrate_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6e4fb53b-539d-4e00-a874-dc8ac39f19f2_calibrate_1039672_39.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1f288f1d-a4ba-4dac-b3b7-f033ea5c1d7b_characterizeImage_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1f288f1d-a4ba-4dac-b3b7-f033ea5c1d7b_characterizeImage_1039672_39.stderr parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 089bfcd56525e9242f1343e0172da518 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/75d93b83-31d6-42d5-8d3c-ac0378db0874_calibrate_1039674_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/75d93b83-31d6-42d5-8d3c-ac0378db0874_calibrate_1039674_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:45 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:45 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb640b13808bd383b94369b5f24dc73 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 7c19f25ad86b9f59bb86013c446838c0 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7073cbbe-2b43-4798-9033-a68082406c6f_calibrate_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7073cbbe-2b43-4798-9033-a68082406c6f_calibrate_1039769_48.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/77e4b5ac-32c4-42c3-bed8-b3f9eeef0d20_characterizeImage_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/77e4b5ac-32c4-42c3-bed8-b3f9eeef0d20_characterizeImage_1039769_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:46 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:46 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11b082596c6c01c0cb30b2d3b320ebbe with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 51874d72e90a7210bf95e8ad36b68264 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/2848f74b-69c4-440f-9f83-f5ef131f7697_calibrate_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/2848f74b-69c4-440f-9f83-f5ef131f7697_calibrate_1039768_8.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f4902d84-db08-4ebc-8066-036d394cd089_characterizeImage_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f4902d84-db08-4ebc-8066-036d394cd089_characterizeImage_1039768_8.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dad5a2215d44227c1f701a7ac5c16a02 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 6c104974e4031dcbbc83d99c8e1916f0 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/699d5c66-a6e2-4a69-b5f8-0cc4c1d7120d_calibrate_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/699d5c66-a6e2-4a69-b5f8-0cc4c1d7120d_calibrate_1039768_18.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b90f7d97-08a2-41d5-ab83-b718ae82e472_characterizeImage_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b90f7d97-08a2-41d5-ab83-b718ae82e472_characterizeImage_1039768_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:47 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:47 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a973d902d7ed6a5f308b69dee725cea4 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 374a8b1363c3e5be0f67a96eaf4d515e parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9122a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/abf55b8f-0e19-4bfb-9902-885399f24f4b_calibrate_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/abf55b8f-0e19-4bfb-9902-885399f24f4b_calibrate_1039768_59.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d8fa4c85-c971-4ae9-8367-0fb8a6bbd6dd_characterizeImage_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d8fa4c85-c971-4ae9-8367-0fb8a6bbd6dd_characterizeImage_1039768_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:48 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:48 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642cbb134df64f0c7c50903cf68b9864 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash bd0a75906902aaa7101781374d90acdd parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/55a38d70-5476-433e-bf9b-d14baf363d66_calibrate_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/55a38d70-5476-433e-bf9b-d14baf363d66_calibrate_1039672_57.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/eb70b5c2-7ebe-486c-93a0-d8fd01399c0a_characterizeImage_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/eb70b5c2-7ebe-486c-93a0-d8fd01399c0a_characterizeImage_1039672_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:49 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33eb4bd331b14d0ef818fb60764157e6 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/414e30f3-eccd-4adb-933f-de2de2619e0d_calibrate_1039674_36.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/414e30f3-eccd-4adb-933f-de2de2619e0d_calibrate_1039674_36.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c08c327641201afa55ffa2ce817eb9fd with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash af4fe94591884edb1605b82265174114 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9391c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/79ebdfaf-a3b3-4528-8956-556b04330997_calibrate_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/79ebdfaf-a3b3-4528-8956-556b04330997_calibrate_1039674_10.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a5748f1c-92f4-44f8-8a1d-27914fde4a9e_characterizeImage_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/a5748f1c-92f4-44f8-8a1d-27914fde4a9e_characterizeImage_1039674_10.stderr parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4e029e3638bed17fcac30be46f1d1a with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 4f2010490dcb6491b3614220aabc8684 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1dce5dfe-18ac-43f1-8a62-da0f42cd0d72_calibrate_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1dce5dfe-18ac-43f1-8a62-da0f42cd0d72_calibrate_1039672_59.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92c8ad5e-194a-48ed-aaec-84f82dc2b253_characterizeImage_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92c8ad5e-194a-48ed-aaec-84f82dc2b253_characterizeImage_1039672_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:49 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53a2f08d22945d0200d968fc3297a0cc with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 38e143e8f870db7d29715bbcab36feaa parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e279c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f12ce55d-9e1e-4738-a65e-8296a55c6c04_calibrate_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f12ce55d-9e1e-4738-a65e-8296a55c6c04_calibrate_1039770_4.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a198abf0-275c-4962-8827-a2885e602d67_characterizeImage_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a198abf0-275c-4962-8827-a2885e602d67_characterizeImage_1039770_4.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ea6cab135f276c3ae1e27d81dde56a with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash 6e1639eb3e0f45d441644539d5d713e6 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/56e56363-0726-4393-b59b-d9333ba68c46_calibrate_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/56e56363-0726-4393-b59b-d9333ba68c46_calibrate_1039673_43.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c5e3f89-289b-40c6-a3a4-2770e0e4d2bd_characterizeImage_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c5e3f89-289b-40c6-a3a4-2770e0e4d2bd_characterizeImage_1039673_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:50 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d532bdfa6e0202aa6237e207a6d4ba4d with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ec67b3c1-c5ba-4824-96aa-1078619fe4b9_calibrate_1039672_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ec67b3c1-c5ba-4824-96aa-1078619fe4b9_calibrate_1039672_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:50 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7628a627fc0246e2bcaea1f6edcf45d with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash bfddfe2e9c0d1e990cc504a290c4978e parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/2655ee26-2876-4cbd-8fa6-0672a68f4b8a_calibrate_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/2655ee26-2876-4cbd-8fa6-0672a68f4b8a_calibrate_1039673_8.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3e15da0f-a1bd-4d81-8fb4-5662b522d70a_characterizeImage_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/3e15da0f-a1bd-4d81-8fb4-5662b522d70a_characterizeImage_1039673_8.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc11a859a808e53b2c32ef0b65e824fe with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/199981ba-e442-4994-b11b-1acf5c5d7cf5_calibrate_1039673_7.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/199981ba-e442-4994-b11b-1acf5c5d7cf5_calibrate_1039673_7.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a90069540e6585b69ae81549b8e60ea with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash 8d027f0d89879f82f2167e1418f03556 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d0b55f6-b0b2-4686-ba61-7c43f966b1c2_calibrate_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d0b55f6-b0b2-4686-ba61-7c43f966b1c2_calibrate_1039674_41.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5b180f17-3673-4cfe-a7e5-c9698babd0ba_characterizeImage_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5b180f17-3673-4cfe-a7e5-c9698babd0ba_characterizeImage_1039674_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:51 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9410a9d2b87b964525f047174a401e29 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash e487b1bd61874cae5aff908696e70e85 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/62519e82-dcb8-440a-b073-990ebc5ac4aa_calibrate_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/62519e82-dcb8-440a-b073-990ebc5ac4aa_calibrate_1039673_57.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4be71972-336f-4d64-8f47-e5b5b76178dd_characterizeImage_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4be71972-336f-4d64-8f47-e5b5b76178dd_characterizeImage_1039673_57.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 377e6449ac1c26728f3937a2ba5b9fd6 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/3b518a7c-2590-4add-8cb2-b0054dd0d31f_calibrate_1039674_40.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/3b518a7c-2590-4add-8cb2-b0054dd0d31f_calibrate_1039674_40.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bac1841fc1661b25ef598eea8c3a0ad with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash e6a2bf1ee7835842d1ad07f3fde76d4a parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/515a9f31-8138-48f1-9bf7-49155084aec7_calibrate_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/515a9f31-8138-48f1-9bf7-49155084aec7_calibrate_1039674_57.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5238ee21-b584-45d0-a0a5-59290cfdad2f_characterizeImage_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5238ee21-b584-45d0-a0a5-59290cfdad2f_characterizeImage_1039674_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:53 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b1d8d51567186b30d96bb93164d5144 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/38cc02dd-7066-4fb1-b7bd-13cbca40f02a_calibrate_1039673_18.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/38cc02dd-7066-4fb1-b7bd-13cbca40f02a_calibrate_1039673_18.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df2a99ccf34b2b7a39205adcedcc50bc with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash a4579df4135a335daa5197425c29d5a5 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/0521abb8-a06b-40b3-a5c6-5d3ed75c995a_calibrate_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/0521abb8-a06b-40b3-a5c6-5d3ed75c995a_calibrate_1039768_15.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f87ffc45-7aa8-4471-9546-2665217375b8_characterizeImage_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f87ffc45-7aa8-4471-9546-2665217375b8_characterizeImage_1039768_15.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07c0699b1f45c0676011935555571cd9 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash d7db5c203bc3b2804a8f61726933d8c2 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/caec8f16-76fb-4fd7-9b3d-40647d5b7dd2_calibrate_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/caec8f16-76fb-4fd7-9b3d-40647d5b7dd2_calibrate_1039674_51.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/012f55c1-b670-4549-878e-b5397649d8c1_characterizeImage_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/012f55c1-b670-4549-878e-b5397649d8c1_characterizeImage_1039674_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:53 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ece9b713f64fb969206a1fc295d56b5 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 1464be4be0014f31a02d5800fb019114 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/196428fc-a145-48a3-b9bf-80cacb3733b5_calibrate_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/196428fc-a145-48a3-b9bf-80cacb3733b5_calibrate_1039770_3.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4bbcad05-6c57-4f15-935e-18a7352296ee_characterizeImage_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4bbcad05-6c57-4f15-935e-18a7352296ee_characterizeImage_1039770_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:54 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e632633f14463af31324a34d0a4a3477 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash d678953ad4cec5bf54efe2d8f4627aae parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/41dc1f33-bed4-4ca3-b669-92cf8c2dd697_calibrate_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/41dc1f33-bed4-4ca3-b669-92cf8c2dd697_calibrate_1039674_43.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/89bd3e8f-8c7c-4243-aa06-4ec85d86807e_characterizeImage_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/89bd3e8f-8c7c-4243-aa06-4ec85d86807e_characterizeImage_1039674_43.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fb60b54aa7a6fb3f5787460ee725295 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/58ba113c-8281-4c58-a899-fe66658994ec_calibrate_1039672_27.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/58ba113c-8281-4c58-a899-fe66658994ec_calibrate_1039672_27.stderr parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f402ea8a8d5f07f84f588e0e3aa220eb with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 08123c6bb892c2ad3b53ff99c5fa6e55 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/acf7e7fe-4f50-49c4-a6fd-0cf9e67a55e2_calibrate_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/acf7e7fe-4f50-49c4-a6fd-0cf9e67a55e2_calibrate_1039672_11.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b070a01f-05a9-4f40-8ff5-bbd41eb296d1_characterizeImage_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b070a01f-05a9-4f40-8ff5-bbd41eb296d1_characterizeImage_1039672_11.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ed1d2147301547cf6470477aa5f8bfe with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 49f7ea9c35e984fde3b2e4dd836dcf61 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/d5d618da-43f5-4aa7-abd0-e2ec176f126f_calibrate_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/d5d618da-43f5-4aa7-abd0-e2ec176f126f_calibrate_1039673_33.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9e5a41da-d991-42ef-a923-b4cdf7c53559_characterizeImage_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/9e5a41da-d991-42ef-a923-b4cdf7c53559_characterizeImage_1039673_33.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c9472c9ec54ef3ada4897ac7e3a533c with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/aaf26a78-4fde-403e-b667-0aa592e6634f_calibrate_1039673_27.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/aaf26a78-4fde-403e-b667-0aa592e6634f_calibrate_1039673_27.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a71d5b9eb269b14cddd3eaa479a0259 with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash e41553aebe8cfa3d66e805ee7b79f0b3 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0838d5c9-bdc5-431e-a1f8-96b1621ff74d_calibrate_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0838d5c9-bdc5-431e-a1f8-96b1621ff74d_calibrate_1039672_53.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/77221497-5df3-495f-bc6c-8bc6674d1e9e_characterizeImage_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/77221497-5df3-495f-bc6c-8bc6674d1e9e_characterizeImage_1039672_53.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0393df87e3f9812cc03e6bea3e533ab with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 0aa003d2fdfe67e2f47355e77fb86a49 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9109a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3a8b6e4e-e937-4ab0-a89d-5b8f25d663e9_calibrate_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3a8b6e4e-e937-4ab0-a89d-5b8f25d663e9_calibrate_1039673_6.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8751ab07-8308-4ddb-8096-d7274e7ec00c_characterizeImage_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8751ab07-8308-4ddb-8096-d7274e7ec00c_characterizeImage_1039673_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:54 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35a41a47b152c46f23622286e99530cd with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 04c1182e3afbe02516d595dfbf61cd7c parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d9377a67-dbbf-4481-93d8-5da66691b0cd_calibrate_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d9377a67-dbbf-4481-93d8-5da66691b0cd_calibrate_1039674_46.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9c499b82-717e-4b9d-b93c-981e452e0411_characterizeImage_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9c499b82-717e-4b9d-b93c-981e452e0411_characterizeImage_1039674_46.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4484e12fc73290f25303f96fadc9e2c6 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 8e5faa32b382c6cbefc616d747fe443e parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/2ea26240-4f08-467a-a7d1-bc2e95e7e8a9_calibrate_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/2ea26240-4f08-467a-a7d1-bc2e95e7e8a9_calibrate_1039769_57.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/bcc9c4a3-8ba5-4ba9-88a9-1601d6a20627_characterizeImage_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/bcc9c4a3-8ba5-4ba9-88a9-1601d6a20627_characterizeImage_1039769_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:55 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae04ad9c622ceeb691d12727bde2f17 with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/1e9b79b1-b1fa-438b-8371-9e6d02ecc787_calibrate_1039674_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/1e9b79b1-b1fa-438b-8371-9e6d02ecc787_calibrate_1039674_42.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee5e8ea2c0e273b33c6034a5497a38c0 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/271c0ae4-0dfc-450d-bf4c-af0d08bffd2e_calibrate_1039672_42.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/271c0ae4-0dfc-450d-bf4c-af0d08bffd2e_calibrate_1039672_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:55 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96462759816b30abb2afd2921f5a4762 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 2831e5d31d27547cd390e4373f808071 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d4bbe42-fba4-4549-b6c0-ec3f920e43c3_calibrate_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d4bbe42-fba4-4549-b6c0-ec3f920e43c3_calibrate_1039672_21.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b90ba4e6-f991-4f03-b5af-55c8724ec8c3_characterizeImage_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b90ba4e6-f991-4f03-b5af-55c8724ec8c3_characterizeImage_1039672_21.stderr parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b52602cfce99969a2299535637954f00 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/11816491-dcca-4207-8859-21b119265be2_calibrate_1039673_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/11816491-dcca-4207-8859-21b119265be2_calibrate_1039673_40.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0caacf0273fdbbe570c4fa5ddba7ac7c with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash d47c9d3dd246c02d422b4b9c710a7fe1 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1bfa6e37-810f-4333-8f86-c2233decab4c_calibrate_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1bfa6e37-810f-4333-8f86-c2233decab4c_calibrate_1039672_32.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9515a74a-9775-4b44-b445-a82d76d1b270_characterizeImage_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9515a74a-9775-4b44-b445-a82d76d1b270_characterizeImage_1039672_32.stderr parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04f9dc0cf3c025d8b2293c298bf98cd0 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash da769b60107f8fcecbe09df638df2162 parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/77fbcba7-f7e7-4da8-af9c-12b110782355_calibrate_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/77fbcba7-f7e7-4da8-af9c-12b110782355_calibrate_1039673_28.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/46e07f1b-f925-4afc-a03f-1b8de454c0e5_characterizeImage_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/46e07f1b-f925-4afc-a03f-1b8de454c0e5_characterizeImage_1039673_28.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06af83fe06c655c18d6fbd7178ffa3fc with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/63baf1b0-066d-4771-be40-140c26eb8ded_calibrate_1039768_31.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/63baf1b0-066d-4771-be40-140c26eb8ded_calibrate_1039768_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:56 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31385ba72d4703e6426c567c5b01b4eb with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 61c76af837b70c64c1eb0d3104fc5f5d parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a8d70291-f947-48fb-8dbc-444e5dc8e443_calibrate_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a8d70291-f947-48fb-8dbc-444e5dc8e443_calibrate_1039768_13.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f0d1b283-0fb9-4b24-be85-19126f71dbe1_characterizeImage_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/f0d1b283-0fb9-4b24-be85-19126f71dbe1_characterizeImage_1039768_13.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a6fc7999f00f203c21c9c0eb00e904d with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash d0e858b81d337fe2a9a4e6f3a83a34f1 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7c3c853c-f9f8-4263-a2df-efc07b08509f_calibrate_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7c3c853c-f9f8-4263-a2df-efc07b08509f_calibrate_1039672_33.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1a39406a-dc14-4b28-bfb5-aa92e55ac7ed_characterizeImage_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1a39406a-dc14-4b28-bfb5-aa92e55ac7ed_characterizeImage_1039672_33.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6848d233f8db78643880f285dc151f2b with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 1c0a2ebae9283419cfdcbd64bc3ea08a parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d96d720c-d861-41e4-b362-71a783e09dc7_calibrate_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d96d720c-d861-41e4-b362-71a783e09dc7_calibrate_1039672_18.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/714d1cd7-42b3-4b6c-94e3-3001262f1b66_characterizeImage_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/714d1cd7-42b3-4b6c-94e3-3001262f1b66_characterizeImage_1039672_18.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2937afc2f697b335307dadb878fc5f with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 33099a19c91e16bb992b436db3ac55e3 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/59559173-e247-47b9-bb3c-c84d6d7e55b6_calibrate_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/59559173-e247-47b9-bb3c-c84d6d7e55b6_calibrate_1039672_1.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/80ae5e2b-51bb-463d-9b76-06194f37d592_characterizeImage_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/80ae5e2b-51bb-463d-9b76-06194f37d592_characterizeImage_1039672_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:56 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe87af87c4ed36fde0f4c87d0fddd5c3 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 30ffab1e743ed8721ba1215d5e902fad parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d79b6233-1a29-4846-a390-e0bcb9a17a3b_calibrate_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d79b6233-1a29-4846-a390-e0bcb9a17a3b_calibrate_1039768_57.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/c12e208f-a8cc-4a4d-a4e8-03c14105dbe2_characterizeImage_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/c12e208f-a8cc-4a4d-a4e8-03c14105dbe2_characterizeImage_1039768_57.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4051cb6367121b018d1c84328157e029 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 5b24a2130c783cd595ef1b24c1b4223e parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f911c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d4943173-d773-410a-844c-d83cccced265_calibrate_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d4943173-d773-410a-844c-d83cccced265_calibrate_1039768_44.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/20cdff64-475f-4928-80d8-1c295aa9e620_characterizeImage_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/20cdff64-475f-4928-80d8-1c295aa9e620_characterizeImage_1039768_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:57 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdba461ba7e1d803857f48539bfe4dc9 with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 1b93d161c1b72cb23a225b35b0735a2b parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ae81b794-ef70-4ccd-b66a-78251ad388f3_calibrate_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ae81b794-ef70-4ccd-b66a-78251ad388f3_calibrate_1039674_17.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/7291f0b1-5734-4d5b-848c-3d52fcc3370d_characterizeImage_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/7291f0b1-5734-4d5b-848c-3d52fcc3370d_characterizeImage_1039674_17.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac115b651f636e4c77b5bbc2c242f35c with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/56d3ace8-1d4e-4211-aa89-841eb1b3dc70_calibrate_1039672_50.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/56d3ace8-1d4e-4211-aa89-841eb1b3dc70_calibrate_1039672_50.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea30bac02e06b3e90a37cb50bc961a2a with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash e358ba9419269efcaa49a166ca360764 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/3342d075-0f6b-494e-85a9-91e2758c108c_calibrate_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/3342d075-0f6b-494e-85a9-91e2758c108c_calibrate_1039672_48.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a7af225-f5a9-4334-9722-3770b7ab9f22_characterizeImage_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a7af225-f5a9-4334-9722-3770b7ab9f22_characterizeImage_1039672_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:57 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b23ade52f300207ca67507424ca668b1 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash a4dc131c646d846495a895fa823cda78 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3814e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c01ec5b2-4295-483b-9cc5-6600c591f9a4_calibrate_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c01ec5b2-4295-483b-9cc5-6600c591f9a4_calibrate_1039673_45.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/275425f6-759e-41ce-99ac-dbe453c9fc60_characterizeImage_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/275425f6-759e-41ce-99ac-dbe453c9fc60_characterizeImage_1039673_45.stderr parsl.dataflow.dflow INFO: Task 1106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48d1a077f5673fe2bb58ebac2b075bda with result from task 1106 parsl.dataflow.dflow DEBUG: Task 1106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1d1fade6-2168-4b8c-bf01-5472027eba2b_calibrate_1039672_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1d1fade6-2168-4b8c-bf01-5472027eba2b_calibrate_1039672_51.stderr parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef9a4f83390781f3187419609f53cd38 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash 5cf976debff7e7120b3f76e357c80b04 parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d691dec1-4a85-4d97-95b1-c37f7740404d_calibrate_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d691dec1-4a85-4d97-95b1-c37f7740404d_calibrate_1039674_3.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da8293d7-6cc7-40e9-982c-fbf265565b42_characterizeImage_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da8293d7-6cc7-40e9-982c-fbf265565b42_characterizeImage_1039674_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:58 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe61226bb3c90eeed680c42be0ee2edb with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/230117a6-7579-4819-b790-cb043c7e3774_calibrate_1039672_40.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/230117a6-7579-4819-b790-cb043c7e3774_calibrate_1039672_40.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f332c215eb4db260f01bcd412514f4e with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash 129a65e363f2ed04d2d0376b88c94c11 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/783e5192-11f6-4f3f-b753-0bdb7148a0ad_calibrate_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/783e5192-11f6-4f3f-b753-0bdb7148a0ad_calibrate_1039674_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6b07db35-e324-4964-b05d-4f07f2dc0ca9_characterizeImage_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/6b07db35-e324-4964-b05d-4f07f2dc0ca9_characterizeImage_1039674_45.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c93f446535928e23a9372941d1d1f45 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 6396de9bdd251d86b3f77a71f7714d50 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8aca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fb3ee3e0-db25-4414-9686-4d0faed72d81_calibrate_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fb3ee3e0-db25-4414-9686-4d0faed72d81_calibrate_1039770_35.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/845c5fe4-159d-4d2b-9c2a-ad8a493112c1_characterizeImage_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/845c5fe4-159d-4d2b-9c2a-ad8a493112c1_characterizeImage_1039770_35.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd50ad8ba0951b4d04e63fb6d1292c03 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 1405bcff8a068a8afd1ae9b52654f769 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1e23e149-e3f8-4ddb-9fec-8663906cd9c6_calibrate_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1e23e149-e3f8-4ddb-9fec-8663906cd9c6_calibrate_1039672_20.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/98ac3ef6-d8f1-43ab-89e9-5f5e314baf07_characterizeImage_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/98ac3ef6-d8f1-43ab-89e9-5f5e314baf07_characterizeImage_1039672_20.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a909d501a4a62c1d93e30f16dc08e081 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/08c9f3e1-7cc5-4a31-9ed0-e64635502a70_calibrate_1039674_23.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/08c9f3e1-7cc5-4a31-9ed0-e64635502a70_calibrate_1039674_23.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be832005c17facf05d8395aff88e451 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 43564120c112f876b35cc1814c9ecfce parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ffef57ce-03c9-4829-b0d5-ab75866cce5f_calibrate_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ffef57ce-03c9-4829-b0d5-ab75866cce5f_calibrate_1039672_3.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/adcb998c-d55d-41a9-98a8-a47f698e39a9_characterizeImage_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/adcb998c-d55d-41a9-98a8-a47f698e39a9_characterizeImage_1039672_3.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd152039c3ec59d467f40c1a4fe6a632 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash 9fbcfd50cdfbe992b767722c4cae944e parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/812099fc-1c3a-4b88-84ff-388faa834e26_calibrate_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/812099fc-1c3a-4b88-84ff-388faa834e26_calibrate_1039673_55.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/255e7c21-ca5c-46c3-8e7e-473ffbb505cb_characterizeImage_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/255e7c21-ca5c-46c3-8e7e-473ffbb505cb_characterizeImage_1039673_55.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9595382aef5c5d99c8270a25b52f2e3 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash 5e07767107ebb0953defaf06e1a920bb parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ac6e8bc5-9597-471c-8202-563d597368bf_calibrate_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ac6e8bc5-9597-471c-8202-563d597368bf_calibrate_1039672_25.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/55a4c813-1264-4e1d-a1dd-5a6badfa273e_characterizeImage_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/55a4c813-1264-4e1d-a1dd-5a6badfa273e_characterizeImage_1039672_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:59 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34edd86224df89cb1b2ee90062c27146 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash a4334fb25e02453becc39c17b2113b81 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5b2f8ca2-3790-4195-bbad-78cce3baf141_calibrate_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5b2f8ca2-3790-4195-bbad-78cce3baf141_calibrate_1039673_46.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/66962144-03ce-45e3-8f5f-a1388231a522_characterizeImage_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/66962144-03ce-45e3-8f5f-a1388231a522_characterizeImage_1039673_46.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cd2db4faa7ac67b66ea86a62097fab2 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash afa0dd30585772157266e3e34714ab19 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b8edb12-eef0-4d74-a78e-5305c98ded4e_calibrate_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b8edb12-eef0-4d74-a78e-5305c98ded4e_calibrate_1039673_60.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b22bad5b-96a6-4b50-bc58-22416922b735_characterizeImage_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b22bad5b-96a6-4b50-bc58-22416922b735_characterizeImage_1039673_60.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f01a9bb8e23eff4598d8ffb7f2d3a781 with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 6b2c949731fe88e09b1b8c48baaff913 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0a69c995-5e8d-4bf0-936c-7e5bac0a1364_calibrate_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0a69c995-5e8d-4bf0-936c-7e5bac0a1364_calibrate_1039674_11.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/200a6d68-4733-4d9b-a9f4-1f772e7c7d3b_characterizeImage_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/200a6d68-4733-4d9b-a9f4-1f772e7c7d3b_characterizeImage_1039674_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:40:59 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c969f7ac56af3655107c306e4c1e422 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 4bbe564406d23808faea0e563c7bf5fa parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/107de102-9f0c-45cd-ae78-ab4aaf0421a8_calibrate_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/107de102-9f0c-45cd-ae78-ab4aaf0421a8_calibrate_1039673_42.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6cd0d9f4-6e19-4bf4-85c9-f59fa7e34ff7_characterizeImage_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6cd0d9f4-6e19-4bf4-85c9-f59fa7e34ff7_characterizeImage_1039673_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:01 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3db68f8f0e62a608118fd2f68d40130 with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash f9136f42c35227d03289858ca276484d parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a894960d-0356-4d3f-94f9-8b10c12ae5ce_calibrate_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a894960d-0356-4d3f-94f9-8b10c12ae5ce_calibrate_1039672_28.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/033d07da-e218-46f4-b1d1-c8460d233f4e_characterizeImage_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/033d07da-e218-46f4-b1d1-c8460d233f4e_characterizeImage_1039672_28.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 240fc457750c072a6de124e799eb7319 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/e8dfc5a1-6586-4d4a-9a10-ea0ae89b431a_calibrate_1039672_36.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/e8dfc5a1-6586-4d4a-9a10-ea0ae89b431a_calibrate_1039672_36.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b74d840327f4cad1a5ff43ce89e33de0 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 73fe4a359cba7e878bf2c2b9cf61b9c7 parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/08ec79e3-5fb0-4d82-9dbd-c4bc82c3a534_calibrate_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/08ec79e3-5fb0-4d82-9dbd-c4bc82c3a534_calibrate_1039672_29.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b8b20da2-ba2f-4ea3-ac3b-ad0ceba67074_characterizeImage_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/b8b20da2-ba2f-4ea3-ac3b-ad0ceba67074_characterizeImage_1039672_29.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03edac2a7c16ab46dcdb21712cea95af with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash ffd0464dac16bdb987dd39ad439885a9 parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/e0c3929b-23b3-46ea-a97c-378c8bc500db_calibrate_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/e0c3929b-23b3-46ea-a97c-378c8bc500db_calibrate_1039674_55.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93dbc31a-fb7b-4e15-a8c3-a1ad05c89ef3_characterizeImage_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/93dbc31a-fb7b-4e15-a8c3-a1ad05c89ef3_characterizeImage_1039674_55.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36a42f58f0cef3b6c213c04c4f3204e0 with result from task 238 parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:01 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 4fcdfe57137ffe176d9dbf16bc64070e parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4e47a799-a116-43c9-a1ab-46bc0020f2ac_calibrate_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4e47a799-a116-43c9-a1ab-46bc0020f2ac_calibrate_1039674_47.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/75cac794-0f22-46e9-aa42-5cadc8f9ddaf_characterizeImage_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/75cac794-0f22-46e9-aa42-5cadc8f9ddaf_characterizeImage_1039674_47.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 215fd42eafaf1aa3e3f763745dafd8d4 with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 25fb71c8b64bd670123c66582b348e1f parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c2485504-1cdc-4491-8c28-becb826a0b05_calibrate_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c2485504-1cdc-4491-8c28-becb826a0b05_calibrate_1039674_9.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/15517832-c596-4197-9d0e-3ac701f31355_characterizeImage_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/15517832-c596-4197-9d0e-3ac701f31355_characterizeImage_1039674_9.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7920f813ae7174fa5c48a98e124d8d3 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 4fc727e22c8c77863a46945bd6812df2 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/a032fb44-b37f-48bd-9e4f-01e698bab973_calibrate_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/a032fb44-b37f-48bd-9e4f-01e698bab973_calibrate_1039673_56.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b9804e2f-8622-430f-ba67-416e1a44d801_characterizeImage_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/b9804e2f-8622-430f-ba67-416e1a44d801_characterizeImage_1039673_56.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 543dae1fc2130a44a6ad018121169742 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8306abd2-ad7a-4a49-9543-e8ab20a7c50d_calibrate_1039673_36.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8306abd2-ad7a-4a49-9543-e8ab20a7c50d_calibrate_1039673_36.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e1c938fed79ae7130ac548c94598b6 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 5123693aaf77d98a77035f429ce91e5a parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/b5b7a097-aab5-47e2-b9eb-3c11156d19ce_calibrate_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/b5b7a097-aab5-47e2-b9eb-3c11156d19ce_calibrate_1039672_45.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a2230e5b-c1c2-4e01-9c54-b122072027b1_characterizeImage_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/a2230e5b-c1c2-4e01-9c54-b122072027b1_characterizeImage_1039672_45.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a87367264c9de814e82919d3ef68759 with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 02e81a241116743fd2e2974982693abf parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/915a3f0b-63df-4d4a-942f-d70551ee1662_calibrate_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/915a3f0b-63df-4d4a-942f-d70551ee1662_calibrate_1039770_16.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0129c161-1a6d-470d-8482-6b46247da61a_characterizeImage_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0129c161-1a6d-470d-8482-6b46247da61a_characterizeImage_1039770_16.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8134c62cac0bbef1e740930c513727bd with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash e5608918ef5624392e5e30aedbeeeff9 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d37f3a1-9685-41b6-beaf-ff6c8d9d1aa1_calibrate_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d37f3a1-9685-41b6-beaf-ff6c8d9d1aa1_calibrate_1039674_27.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/20b74a1a-a52c-4de2-920e-6461d2da2ebd_characterizeImage_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/20b74a1a-a52c-4de2-920e-6461d2da2ebd_characterizeImage_1039674_27.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38681c4aa6502d0032dba0146001b71d with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 7972a238dcef9d434291d33fa1306490 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0ea1867f-d42a-4c44-9c4c-e958a1ef0972_calibrate_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0ea1867f-d42a-4c44-9c4c-e958a1ef0972_calibrate_1039674_35.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f00c1689-38b3-4cde-a5f6-5eb17c304435_characterizeImage_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f00c1689-38b3-4cde-a5f6-5eb17c304435_characterizeImage_1039674_35.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f6b9a128747c74b8cc50aac6831aaf with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash e86ed5f99fe67410c1a4ca93dca11cbb parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a6394304-6a53-44e8-92a9-294555ebb1b5_calibrate_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a6394304-6a53-44e8-92a9-294555ebb1b5_calibrate_1039674_54.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/3ab236f9-7f6d-45c6-b418-2df4fb754383_characterizeImage_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/3ab236f9-7f6d-45c6-b418-2df4fb754383_characterizeImage_1039674_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:02 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd0a510695e662afb1906ac957a0e72 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 635d96103041dbac363edf2a051f20fa parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/beae0f4a-89ec-4c64-8b0f-85041ee3ce04_calibrate_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/beae0f4a-89ec-4c64-8b0f-85041ee3ce04_calibrate_1039769_49.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/055541bd-aa93-4e36-b7f8-90fba62dd471_characterizeImage_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/055541bd-aa93-4e36-b7f8-90fba62dd471_characterizeImage_1039769_49.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a89f1efb87f04f46684dfc6d961b03a with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash db8e98995ea9b55957b6fd69cc9a6610 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/1602b8ba-cf64-47fb-99f8-00f1d982227c_calibrate_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/1602b8ba-cf64-47fb-99f8-00f1d982227c_calibrate_1039769_11.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/33f37876-183e-4734-a2c3-8f8d9440ca93_characterizeImage_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/33f37876-183e-4734-a2c3-8f8d9440ca93_characterizeImage_1039769_11.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4635b447660e22ab0b6357cb579a9bd with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/924a0756-0b25-4e91-bb54-b5c5aa526ece_calibrate_1039674_21.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/924a0756-0b25-4e91-bb54-b5c5aa526ece_calibrate_1039674_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:04 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da8b53625ca5a1298070775e852fb5a4 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 3cbdbf6aeb6f140524356ccedabe6da6 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/04dde255-6a4e-42b1-9c8b-34ca47eb37cd_calibrate_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/04dde255-6a4e-42b1-9c8b-34ca47eb37cd_calibrate_1039672_58.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fec686f0-a0c1-4b03-945c-5eeb600b4575_characterizeImage_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/fec686f0-a0c1-4b03-945c-5eeb600b4575_characterizeImage_1039672_58.stderr parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e90c2f67b5d0e9cdcf93669cfa5cfeb8 with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 501e6e88b942747bf06a665b6a96ad73 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5845820b-2225-4bc1-9d7d-f3c73a2aa41b_calibrate_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5845820b-2225-4bc1-9d7d-f3c73a2aa41b_calibrate_1039770_13.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a0750179-4c89-4978-8e0c-472b80d155aa_characterizeImage_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a0750179-4c89-4978-8e0c-472b80d155aa_characterizeImage_1039770_13.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa57a7a23038a5ab6293134eec52dc5 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash ac74010eb3311ecec116580fe28391a2 parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5cb0bc4-3b02-4d58-883c-a4e52d1a472b_calibrate_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5cb0bc4-3b02-4d58-883c-a4e52d1a472b_calibrate_1039768_17.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e2805d44-c061-4d2f-a76a-652ca5f9de27_characterizeImage_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e2805d44-c061-4d2f-a76a-652ca5f9de27_characterizeImage_1039768_17.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db972311f8c01d9d72dda1fabd6542d4 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 67af9cfd2ba260fa67eccb87a26b46b0 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ee695afb-e12b-42f7-a7d0-df7acb53e61d_calibrate_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ee695afb-e12b-42f7-a7d0-df7acb53e61d_calibrate_1039770_42.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/dc9af798-997a-48ad-86b5-1ea947ecad56_characterizeImage_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/dc9af798-997a-48ad-86b5-1ea947ecad56_characterizeImage_1039770_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:04 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adae69e7475cfddbafa7207e09bf92d1 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash e1653f3a4609b9ca35e2ee072f917546 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c3747be-efb1-4475-a002-a478fa70339e_calibrate_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c3747be-efb1-4475-a002-a478fa70339e_calibrate_1039673_38.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/32cbb219-26fe-4b29-9ad6-f4f1dd8017d4_characterizeImage_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/32cbb219-26fe-4b29-9ad6-f4f1dd8017d4_characterizeImage_1039673_38.stderr parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80fa4be791584adc2e3f444dff42ca1 with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b8cb098f-d730-406b-b0a6-bb6f51d197db_calibrate_1039673_39.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b8cb098f-d730-406b-b0a6-bb6f51d197db_calibrate_1039673_39.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 764d08e5b5d3cfa2eb14d397563f9b61 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash 13272baccd105bca7fe649a7fe934cea parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/93682c3d-bea1-478c-982a-67f29812847c_calibrate_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/93682c3d-bea1-478c-982a-67f29812847c_calibrate_1039674_53.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f5f7e280-7b1e-454a-9921-b0d7060f511c_characterizeImage_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/f5f7e280-7b1e-454a-9921-b0d7060f511c_characterizeImage_1039674_53.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8bb46357387d987630f29d8a7c9055d with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash 874d28c58184a3c2f1d16d52462785fa parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dc0a3f30-3df0-4523-b19d-4ee973beb246_calibrate_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dc0a3f30-3df0-4523-b19d-4ee973beb246_calibrate_1039673_53.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4a7572b7-6233-4820-b98f-8792a8aeecb7_characterizeImage_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/4a7572b7-6233-4820-b98f-8792a8aeecb7_characterizeImage_1039673_53.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc05b6b974dfa702b71d0b73264d79d4 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 78a1a1bf87150396b03313a90ef5d438 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b2c355cc-df3d-45d7-95e9-bbefc5954472_calibrate_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b2c355cc-df3d-45d7-95e9-bbefc5954472_calibrate_1039673_4.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/ea323feb-31fb-474b-863e-61fbafe103e4_characterizeImage_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/ea323feb-31fb-474b-863e-61fbafe103e4_characterizeImage_1039673_4.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9078dab436c51d405f2f42e588ce293f with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 0a2cf49aebfad90bc95094d7132dc317 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8e756723-ab97-449f-87c7-18c5ae35dbf4_calibrate_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8e756723-ab97-449f-87c7-18c5ae35dbf4_calibrate_1039672_38.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/17348e82-88bb-4c91-bc41-23e6d2573d5f_characterizeImage_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/17348e82-88bb-4c91-bc41-23e6d2573d5f_characterizeImage_1039672_38.stderr parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6282aea18664149395702b11b8a670c9 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/0e9a2f00-1ba8-4c24-8b30-ef9539eceaa2_calibrate_1039673_50.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/0e9a2f00-1ba8-4c24-8b30-ef9539eceaa2_calibrate_1039673_50.stderr parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691033662c472b9c09790bf85edc8867 with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 768234d9cff5c34bb1403af25c4a8265 parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5b0ff4ca-4d8b-4628-8371-82716fd81468_calibrate_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5b0ff4ca-4d8b-4628-8371-82716fd81468_calibrate_1039769_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/34f51c13-db51-44a3-9255-281c8d5117e7_characterizeImage_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/34f51c13-db51-44a3-9255-281c8d5117e7_characterizeImage_1039769_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:05 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb11a0548fda0d1bdee3c110a760b43d with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash 083d5c3378fe613873e47dfddb732444 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/18f7b382-1d16-4d23-9e0c-55c63a760210_calibrate_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/18f7b382-1d16-4d23-9e0c-55c63a760210_calibrate_1039673_30.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8e3bcfbb-1ea5-473b-870c-8d205ceba8a8_characterizeImage_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/8e3bcfbb-1ea5-473b-870c-8d205ceba8a8_characterizeImage_1039673_30.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f87c465cfee52e77018bfed640b9e862 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash d4b07a18d147109e1619c7eb07859a69 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/11e88854-a8f0-4ee5-b213-19cdb8573325_calibrate_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/11e88854-a8f0-4ee5-b213-19cdb8573325_calibrate_1039769_26.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/73f26676-5f61-49de-8446-421730393e90_characterizeImage_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/73f26676-5f61-49de-8446-421730393e90_characterizeImage_1039769_26.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c51adff1bbf125d600e41e91a1d6f0e with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash a5fddc4a03052d299ac1d569add600a3 parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2149a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b5022e83-068c-409f-9650-8374a5b275e0_calibrate_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b5022e83-068c-409f-9650-8374a5b275e0_calibrate_1039673_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/41dfe033-0c3a-47ae-991f-0ca2510f0555_characterizeImage_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/41dfe033-0c3a-47ae-991f-0ca2510f0555_characterizeImage_1039673_13.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d085c3749f78105b3dbee0511776542 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/89628530-281d-4c7c-b7c6-bd99eebe41ef_calibrate_1039673_22.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/89628530-281d-4c7c-b7c6-bd99eebe41ef_calibrate_1039673_22.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c1054f132b59b2b8fb9ec9cc8a0d508 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c1939e4b-6909-4d01-b881-7fa1d6edb8e8_calibrate_1039674_60.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c1939e4b-6909-4d01-b881-7fa1d6edb8e8_calibrate_1039674_60.stderr parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31be7e0d18c092297d2db84f087589d2 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/610780cd-0a37-4993-a387-8b0574f0f671_calibrate_1039673_21.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/610780cd-0a37-4993-a387-8b0574f0f671_calibrate_1039673_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:06 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05a13fa3ff703f17b81cb422dd74bbfe with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9707bcd3-add0-4450-85d1-0c165ab852d8_calibrate_1039674_4.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9707bcd3-add0-4450-85d1-0c165ab852d8_calibrate_1039674_4.stderr parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4511b07eec5784efe873cb636aaa953c with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash 08d0c4912269c9f37007207ccb0a2bf4 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/36342f51-d29f-4876-81cc-27a69bdc3d5f_calibrate_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/36342f51-d29f-4876-81cc-27a69bdc3d5f_calibrate_1039770_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/00b32d24-1148-4c1d-bae1-245642f3154d_characterizeImage_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/00b32d24-1148-4c1d-bae1-245642f3154d_characterizeImage_1039770_8.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c1793c1a8271af7ed7cde549bc5961b with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 339dd25cb5669927f9d92c09f3cf01df parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f89b00a1-82b8-404e-abd8-797b6d50017d_calibrate_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f89b00a1-82b8-404e-abd8-797b6d50017d_calibrate_1039770_46.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a381b361-50ef-4f46-a91c-25c07b473188_characterizeImage_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a381b361-50ef-4f46-a91c-25c07b473188_characterizeImage_1039770_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 341 parsl.dataflow.strategy DEBUG: Executor multi has 341 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:41:07 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:41:07 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe02373fba432b7a3bf1c9979e615335 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f2c583bc-78b4-4787-baec-588282914a42_calibrate_1039672_56.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f2c583bc-78b4-4787-baec-588282914a42_calibrate_1039672_56.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9d8455c5e2929846d64989b6fe1336 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 6d3ad724b4b8346e3500eb5ee74bfdcd parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/956c5d3b-2054-4d7a-b7f2-b041fcfcc8b2_calibrate_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/956c5d3b-2054-4d7a-b7f2-b041fcfcc8b2_calibrate_1039769_39.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a8f51ce1-7a32-4701-9205-beaad917eb01_characterizeImage_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/a8f51ce1-7a32-4701-9205-beaad917eb01_characterizeImage_1039769_39.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02c6221caecc44ccbf086fcc37ae2c95 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 3d5bc6cae074d33ef898cc37407e392e parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5efd77f6-c915-41d9-8832-395783e38af2_calibrate_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5efd77f6-c915-41d9-8832-395783e38af2_calibrate_1039770_58.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce86a028-76b9-4b1b-b03c-bfaad165fa31_characterizeImage_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce86a028-76b9-4b1b-b03c-bfaad165fa31_characterizeImage_1039770_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:08 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4c78d6e3f105494bfbdd7585620a15c with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 8be31f21df955a899ca2edf9884ee040 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/c1858d6b-dce2-4914-b682-3e70087fe686_calibrate_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/c1858d6b-dce2-4914-b682-3e70087fe686_calibrate_1039672_19.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/40779ba5-7410-48b6-9925-fa82df90fd6a_characterizeImage_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/40779ba5-7410-48b6-9925-fa82df90fd6a_characterizeImage_1039672_19.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b103cce0fc91448102160347fbdb8125 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5056c067-f576-481b-8281-c3fc07618607_calibrate_1039673_48.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5056c067-f576-481b-8281-c3fc07618607_calibrate_1039673_48.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f92f21de8b86940e90ebea1d6c0f4a2 with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/bc6aa2f7-dd30-4180-8ef8-06d0a047b580_calibrate_1039673_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/bc6aa2f7-dd30-4180-8ef8-06d0a047b580_calibrate_1039673_51.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc5a38dd84b175ca799cd62a8e98887e with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ed00b233-18bb-44f9-975c-28110ad0272c_calibrate_1039674_8.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ed00b233-18bb-44f9-975c-28110ad0272c_calibrate_1039674_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:09 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8207795c46eddee15b784bc468470cfa with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 854355745ca48ff5f0dbe48a66a2122c parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/78f988d4-a76f-4e5d-b663-60bde10e73ab_calibrate_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/78f988d4-a76f-4e5d-b663-60bde10e73ab_calibrate_1039673_20.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a6aef048-b551-4641-983a-424224731ff4_characterizeImage_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a6aef048-b551-4641-983a-424224731ff4_characterizeImage_1039673_20.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdff5b380a29ad5a5d621d97eb8892bc with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 9a79ce075325ac2293c2e4ba8ae90008 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1e1a65ad-2d72-4a82-9b6a-fabac9b925ad_calibrate_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1e1a65ad-2d72-4a82-9b6a-fabac9b925ad_calibrate_1039673_44.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cb980859-ba57-4d6b-b4c7-8b32f26031b0_characterizeImage_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cb980859-ba57-4d6b-b4c7-8b32f26031b0_characterizeImage_1039673_44.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb00b0f71d3610bd258e7a9052971b15 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash f04da7ef8174fade9894c303877a36f2 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e2abc373-d8f0-4225-a095-505311163c0e_calibrate_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e2abc373-d8f0-4225-a095-505311163c0e_calibrate_1039769_27.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3879b6d-7d80-4737-be5c-f514d4399162_characterizeImage_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d3879b6d-7d80-4737-be5c-f514d4399162_characterizeImage_1039769_27.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f6e2973b880174a325d4619016483f with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 96c76f597db4e64b1e47744ddf4f3997 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b5945ab2-6328-4c0b-90e6-d430de64c19e_calibrate_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b5945ab2-6328-4c0b-90e6-d430de64c19e_calibrate_1039769_22.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d7a51b0b-deae-4721-ae8d-30422f79a75e_characterizeImage_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d7a51b0b-deae-4721-ae8d-30422f79a75e_characterizeImage_1039769_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:09 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2fcdb309c62804131bbbe3fdf7e41e0 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 40bf88d37c6747bbffc871ec29b768b3 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ca8cea66-4916-4b80-9f71-ea72299b7634_calibrate_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ca8cea66-4916-4b80-9f71-ea72299b7634_calibrate_1039673_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2b44e1fa-c87b-49a2-945d-3e8c2bb9cd00_characterizeImage_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2b44e1fa-c87b-49a2-945d-3e8c2bb9cd00_characterizeImage_1039673_1.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67cf333fe76cbafda1aa0e166f350a12 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash c4d42560968f61af1ada27d7b56af20d parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f910540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7a9b54d9-e024-4c7a-9e21-c10815e3b61f_calibrate_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7a9b54d9-e024-4c7a-9e21-c10815e3b61f_calibrate_1039769_59.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/261b5cbb-43ee-4b43-9694-937411f2c2ef_characterizeImage_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/261b5cbb-43ee-4b43-9694-937411f2c2ef_characterizeImage_1039769_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:09 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2754608e98c16b5b6b5b0c1a5b1035f1 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 9108c1edbb52bcf5bb14353705a61c4a parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/65015bc1-5123-4070-957f-599458fb5f43_calibrate_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/65015bc1-5123-4070-957f-599458fb5f43_calibrate_1039672_10.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f75a726f-c6a1-4e75-98ce-4959fdc7ff41_characterizeImage_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f75a726f-c6a1-4e75-98ce-4959fdc7ff41_characterizeImage_1039672_10.stderr parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2596b6ac74172793272a39db190eca26 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7888171e-db21-481a-bda0-fd8bdd4bddda_calibrate_1039672_46.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7888171e-db21-481a-bda0-fd8bdd4bddda_calibrate_1039672_46.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56bfbc54d02147538e4f2b926600c342 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 0cc41cb4d2ce936fa47bc7c459416e62 parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:10 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ddae20fa-bc2e-4fb0-892e-c03317757554_calibrate_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ddae20fa-bc2e-4fb0-892e-c03317757554_calibrate_1039673_23.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c660a4ee-734f-415a-9209-9ff718735bb4_characterizeImage_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c660a4ee-734f-415a-9209-9ff718735bb4_characterizeImage_1039673_23.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b0d48ae61e22737ffc3bb66bb9812ac with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 0c813a89ffa637dc65bc6e96ba734c1f parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3f7269bf-5246-46d6-b779-2f059a3bb320_calibrate_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3f7269bf-5246-46d6-b779-2f059a3bb320_calibrate_1039769_37.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/865fd6d3-3e50-4c5d-ae7e-5bd20d8478f2_characterizeImage_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/865fd6d3-3e50-4c5d-ae7e-5bd20d8478f2_characterizeImage_1039769_37.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 657b55ba03a2c06d63e8464e851a47e7 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash d7dc73b76f2cffacb7e0dbd4ddcb6963 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5609b5bc-cda4-45ca-8c49-2ee7d9acb754_calibrate_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5609b5bc-cda4-45ca-8c49-2ee7d9acb754_calibrate_1039769_50.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/94cc0521-e8c9-483c-bb2b-6de986065c30_characterizeImage_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/94cc0521-e8c9-483c-bb2b-6de986065c30_characterizeImage_1039769_50.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43108c18e8f2767d1cfadfbd81f6871a with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash e825a32976253ec5e662ad8f292ee195 parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/83deedfc-386e-4b43-a5c2-6de1d8689ab8_calibrate_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/83deedfc-386e-4b43-a5c2-6de1d8689ab8_calibrate_1039769_1.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/816d6fa7-ed55-45f7-ba9f-0526f409dbff_characterizeImage_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/816d6fa7-ed55-45f7-ba9f-0526f409dbff_characterizeImage_1039769_1.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2f17e3dacbf86240600183bcd034cba with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash 2438d343de4ebc9d1b71ab4ce1814817 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ce3bbb42-050b-4dab-9db4-3407080ca5c7_calibrate_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ce3bbb42-050b-4dab-9db4-3407080ca5c7_calibrate_1039770_26.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/22fd8256-859a-4aa1-91b7-c1c3a269023e_characterizeImage_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/22fd8256-859a-4aa1-91b7-c1c3a269023e_characterizeImage_1039770_26.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00aef9c3ae64416d510f70bed59147ab with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 2e5970dd00c66db0799a71de84bc1fea parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7b928ae6-f61c-4c0c-912c-1a6311db7d74_calibrate_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7b928ae6-f61c-4c0c-912c-1a6311db7d74_calibrate_1039769_60.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e2ae242a-59ea-4f72-a28e-9531b264d2c6_characterizeImage_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e2ae242a-59ea-4f72-a28e-9531b264d2c6_characterizeImage_1039769_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:11 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4543b86129efbb377ad2762dcce31dd with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 2afdb30ede318604a38562bcf5cc8090 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4eda5277-8d02-4589-bca1-8fd00aaecff6_calibrate_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4eda5277-8d02-4589-bca1-8fd00aaecff6_calibrate_1039674_14.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/96bec2ad-eb1b-4212-9345-6dcd4a4b3220_characterizeImage_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/96bec2ad-eb1b-4212-9345-6dcd4a4b3220_characterizeImage_1039674_14.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c178e60c53e5ef13086774732179b260 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 47c81cbd4785c4cede7ee1ef388722ee parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/5b302642-a51e-4d6c-9553-b8518f9f3de4_calibrate_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/5b302642-a51e-4d6c-9553-b8518f9f3de4_calibrate_1039674_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/16a616f3-f2a4-4c62-b02d-c27cdc449e3f_characterizeImage_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/16a616f3-f2a4-4c62-b02d-c27cdc449e3f_characterizeImage_1039674_56.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f7b3c4e452b369b22039683eaa33f83 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash bcc9b13419f6d289fe44dab545d04f28 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4210b8ef-c268-4857-8b68-cd3a78d40390_calibrate_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4210b8ef-c268-4857-8b68-cd3a78d40390_calibrate_1039769_45.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/40f3aa90-20a5-4462-adc1-0285c2a9dbdf_characterizeImage_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/40f3aa90-20a5-4462-adc1-0285c2a9dbdf_characterizeImage_1039769_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:11 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9a5b7fa57eebcbef4d0bc75658a6641 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fd480e72-8a81-4456-814e-4640b5852389_calibrate_1039768_43.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fd480e72-8a81-4456-814e-4640b5852389_calibrate_1039768_43.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a16d2bd3c766c3871bcc178d139adfd with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash c5799d96372fb82be135644af35b2d11 parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/787c612e-4cc7-47d7-9d82-51b08c9bab17_calibrate_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/787c612e-4cc7-47d7-9d82-51b08c9bab17_calibrate_1039768_22.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/efa506cc-28b4-457f-b7b9-b1e7b6b473bc_characterizeImage_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/efa506cc-28b4-457f-b7b9-b1e7b6b473bc_characterizeImage_1039768_22.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5315936f6ee6e677d03fe2b2c9992228 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 2f91b46fb72128236acb3983879f622f parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/80b11c93-7eff-46aa-93de-92c2e8543c0f_calibrate_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/80b11c93-7eff-46aa-93de-92c2e8543c0f_calibrate_1039672_22.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a9a15e4-2175-4f7f-92d6-4654a2a03473_characterizeImage_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3a9a15e4-2175-4f7f-92d6-4654a2a03473_characterizeImage_1039672_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:12 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e544aa688808e4abe1e3bab86c947396 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash dba8adfe7d4675a9af1d5b733ec659bf parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9679cd5c-2bdd-49e2-b8e4-b442b3504876_calibrate_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9679cd5c-2bdd-49e2-b8e4-b442b3504876_calibrate_1039674_7.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e5f1c46a-1fb9-463a-b01e-00927938397e_characterizeImage_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e5f1c46a-1fb9-463a-b01e-00927938397e_characterizeImage_1039674_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:12 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574e6fb5e4bdf9b6bcd561e45156e7ec with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash efc4e222916f89f440cff65af72203c2 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/aef789c7-58d6-4956-9fb4-7e4ac8752234_calibrate_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/aef789c7-58d6-4956-9fb4-7e4ac8752234_calibrate_1039768_36.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/00117804-6b22-4003-ac41-6070cc9375c6_characterizeImage_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/00117804-6b22-4003-ac41-6070cc9375c6_characterizeImage_1039768_36.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98aeefe02e253befd74663dbe6c52c8a with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash 03eec8517a7072a5775bb97cfbdf608a parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/38ce0945-0a61-4639-8798-4769578928fa_calibrate_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/38ce0945-0a61-4639-8798-4769578928fa_calibrate_1039672_43.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/654fe64a-de3f-4fa8-bee5-ee86f2a05f2f_characterizeImage_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/654fe64a-de3f-4fa8-bee5-ee86f2a05f2f_characterizeImage_1039672_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:13 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce3a929d38b045d78db6057719dfa37 with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0754b61d-dcd0-45fc-9b4c-7ead0c64659d_calibrate_1039674_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0754b61d-dcd0-45fc-9b4c-7ead0c64659d_calibrate_1039674_38.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b33a8fcc635f82d0c225a2082e3b1a06 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 7853275b2d1849d0bf546fab35a0da97 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a4281c0b-9ccd-4d1e-8d2c-93dcedeb72d8_calibrate_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a4281c0b-9ccd-4d1e-8d2c-93dcedeb72d8_calibrate_1039768_41.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bd0c07ce-6030-44d5-b346-a427d392bd0a_characterizeImage_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bd0c07ce-6030-44d5-b346-a427d392bd0a_characterizeImage_1039768_41.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35af38b53bc15e027208170e032ab83 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash badcc86db2cf4e23926c2ff97d8266ad parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fac0cfd4-fa40-4a40-9693-54045fdab61e_calibrate_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fac0cfd4-fa40-4a40-9693-54045fdab61e_calibrate_1039672_8.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/30b4eeeb-c111-4ac4-9a31-ced76b9c394b_characterizeImage_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/30b4eeeb-c111-4ac4-9a31-ced76b9c394b_characterizeImage_1039672_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:13 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 105b40d176936c9b538a5a17be8835a4 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 7e1066c658d39a8ef67e07212e422f5c parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6b28b54a-df9b-4efa-acca-590d5b421300_calibrate_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6b28b54a-df9b-4efa-acca-590d5b421300_calibrate_1039770_37.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54397817-d736-44a2-89fa-85848addbacf_characterizeImage_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54397817-d736-44a2-89fa-85848addbacf_characterizeImage_1039770_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:14 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d60dda7aead96399b1730b80ba9d1c06 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash 26c0d28e240d762e4fd844a82de4776d parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/b6d127ab-5b87-40a9-9fc5-876c3890f2bb_calibrate_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/b6d127ab-5b87-40a9-9fc5-876c3890f2bb_calibrate_1039768_33.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/12a683a3-de3f-4d29-ac3a-5973f45406bd_characterizeImage_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/12a683a3-de3f-4d29-ac3a-5973f45406bd_characterizeImage_1039768_33.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d421a6930aacbb5d0de7cb6eb4b87c7 with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 80027248b18e854dbf96717f36bc4e0e parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8986bb68-d012-4d24-9626-88c17dcf3780_calibrate_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8986bb68-d012-4d24-9626-88c17dcf3780_calibrate_1039770_14.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/38d1743e-6422-4bb9-bebb-3beef95ad9fc_characterizeImage_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/38d1743e-6422-4bb9-bebb-3beef95ad9fc_characterizeImage_1039770_14.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1430c5af4f15b9881faa7fa62830b668 with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 383325fcecc36f4c4a1b80e73ee66bbf parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ce5204a3-4db0-4c9e-9316-a3c7d7a9f198_calibrate_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ce5204a3-4db0-4c9e-9316-a3c7d7a9f198_calibrate_1039768_39.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a0ad58e-0ce3-4ef0-a98a-0b672096ba8f_characterizeImage_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a0ad58e-0ce3-4ef0-a98a-0b672096ba8f_characterizeImage_1039768_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:14 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75fa3a4ae7eaa16bf9d71267db8c2f5b with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6c582ab1-5008-42ba-aaaf-1c7333ee41bc_calibrate_1039672_15.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6c582ab1-5008-42ba-aaaf-1c7333ee41bc_calibrate_1039672_15.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069dd271893719e691c7eff4fefc1ac3 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 29e77eb43b027702ab5e1803e5061205 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c5dbc536-0f16-4df4-b3d4-4e987060a104_calibrate_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c5dbc536-0f16-4df4-b3d4-4e987060a104_calibrate_1039674_34.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0ac3a32b-26bc-4e46-8a36-9ebc7a7c3ad1_characterizeImage_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0ac3a32b-26bc-4e46-8a36-9ebc7a7c3ad1_characterizeImage_1039674_34.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbcad1e2b1002dbd3b10c76eee33ee29 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/41b196af-4aed-4828-9ea9-86360adb9724_calibrate_1039673_25.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/41b196af-4aed-4828-9ea9-86360adb9724_calibrate_1039673_25.stderr parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4658391f3241a4dc6328941d74605e5 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash 60a069941ee20cebc6360f2f2fab2f0a parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2162a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fa85f0b6-aaa0-413c-87cf-78117b0396fb_calibrate_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fa85f0b6-aaa0-413c-87cf-78117b0396fb_calibrate_1039674_59.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/353b793a-8fea-43ae-8249-b45dc8d0bbc3_characterizeImage_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/353b793a-8fea-43ae-8249-b45dc8d0bbc3_characterizeImage_1039674_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:15 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:15 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2dfb8d833966e1406004313d51b931d with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash d335742e295d11c1a9494000cdc82284 parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d456d604-ebc1-45d6-838c-c0da798eb488_calibrate_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d456d604-ebc1-45d6-838c-c0da798eb488_calibrate_1039672_34.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/238f7a5c-c45e-439a-b82d-94024ca9b3bf_characterizeImage_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/238f7a5c-c45e-439a-b82d-94024ca9b3bf_characterizeImage_1039672_34.stderr parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a9daaf6464e0b42cf673a09e9315d0a with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 38c05b2815df0d8e63dbc2df5227761b parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11267445-0e85-4f3c-8d9d-9bc73d55d930_calibrate_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11267445-0e85-4f3c-8d9d-9bc73d55d930_calibrate_1039770_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ebce88-52e0-4be4-a2ab-bf272f266faf_characterizeImage_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ebce88-52e0-4be4-a2ab-bf272f266faf_characterizeImage_1039770_45.stderr parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e16e244996602ebce5dc190292f022c with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 1eb8380dd6e1160ffcc943a8b84a3c60 parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e380860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c71c3bd0-f352-42a0-b17c-ffbdc04aee33_calibrate_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c71c3bd0-f352-42a0-b17c-ffbdc04aee33_calibrate_1039769_33.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/fecd1772-cf58-4303-a047-7a24348c051c_characterizeImage_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/fecd1772-cf58-4303-a047-7a24348c051c_characterizeImage_1039769_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:16 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e51054587aff0887b01caf58bb60aed with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b43f29bc-77f7-44bb-9380-29f92861e589_calibrate_1039673_49.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b43f29bc-77f7-44bb-9380-29f92861e589_calibrate_1039673_49.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 257ed837650ffe714bd3278b7fb466ae with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f86d8ade-84a3-454e-b0f8-92f6c2dfea23_calibrate_1039673_32.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f86d8ade-84a3-454e-b0f8-92f6c2dfea23_calibrate_1039673_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:16 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27685f719af289331f651b5e11bb4bae with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8504a2fe-97f1-4199-8b06-b3ca72afbda1_calibrate_1039674_30.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8504a2fe-97f1-4199-8b06-b3ca72afbda1_calibrate_1039674_30.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9af800eebbcaa6b11a270ac2e2f61fd9 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash a010efbd2195350471cebd74ae392740 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/20e0518c-d2d8-4820-b264-897430f00754_calibrate_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/20e0518c-d2d8-4820-b264-897430f00754_calibrate_1039768_48.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/256e0038-370b-464f-9f1c-b533b9903a1d_characterizeImage_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/256e0038-370b-464f-9f1c-b533b9903a1d_characterizeImage_1039768_48.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee842016914c67cfd88d5683bd4dc85f with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3348d4a-f326-484a-ad13-e5e3c5289d24_calibrate_1039673_29.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3348d4a-f326-484a-ad13-e5e3c5289d24_calibrate_1039673_29.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a068ace008da661cc8622162c2d6001e with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1ddeb693-f6d4-4038-8f05-e649341a940f_calibrate_1039672_35.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1ddeb693-f6d4-4038-8f05-e649341a940f_calibrate_1039672_35.stderr parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddcd73052b40c6d1e46da5f6da9c9cd0 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 90f57e67e5cdd8f52680847ac2fe9ab0 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e075d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2d1322f7-05e7-4732-ab31-3477803c0f24_calibrate_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2d1322f7-05e7-4732-ab31-3477803c0f24_calibrate_1039674_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e394d4ed-321a-4e22-aea1-21ced761e996_characterizeImage_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e394d4ed-321a-4e22-aea1-21ced761e996_characterizeImage_1039674_32.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc3f06337dbba2e36c095587afb8e9f6 with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash df5fec5f9132827ea558d21d7cb6385b parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/26a7df34-713a-40df-9fe2-986bf7fa19da_calibrate_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/26a7df34-713a-40df-9fe2-986bf7fa19da_calibrate_1039672_6.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9d2f316b-14a5-42ee-a45a-4bd101a6b5b7_characterizeImage_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9d2f316b-14a5-42ee-a45a-4bd101a6b5b7_characterizeImage_1039672_6.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ac7bf9d62aacf1cc46cba355f34f532 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash bb63e5c84fe6d41610643b64503ef58e parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3da54457-054f-45ca-b420-0db7f6d650c7_calibrate_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3da54457-054f-45ca-b420-0db7f6d650c7_calibrate_1039769_3.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/06836a71-92e3-47ea-96b0-d77bb61b78ca_characterizeImage_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/06836a71-92e3-47ea-96b0-d77bb61b78ca_characterizeImage_1039769_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:17 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b2dccf709a8e2b88f37d57029a21045 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ccba0100-9ff6-4a84-9387-f4f3ce39fd57_calibrate_1039770_29.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ccba0100-9ff6-4a84-9387-f4f3ce39fd57_calibrate_1039770_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:17 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7c34d13c812b47d789ea717e00c278 with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 0d99691f52d05602a9463b0686c8e09c parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/32101117-f28e-4e7d-9ab3-2fe1c49a566f_calibrate_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/32101117-f28e-4e7d-9ab3-2fe1c49a566f_calibrate_1039769_35.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c3be8e51-6bb3-4f32-aa0c-9328cc6405bf_characterizeImage_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c3be8e51-6bb3-4f32-aa0c-9328cc6405bf_characterizeImage_1039769_35.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1045723deb8227d2e07a016fa972803 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash b53e27341afc4b6480b1bdc0ea42225b parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/81d6fa47-4b81-4e38-9ecf-785c5d0f47ce_calibrate_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/81d6fa47-4b81-4e38-9ecf-785c5d0f47ce_calibrate_1039672_4.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/293ce291-69ec-46ea-8bf3-c9e006097a50_characterizeImage_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/293ce291-69ec-46ea-8bf3-c9e006097a50_characterizeImage_1039672_4.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 870a48b7e79605eed7235b903a276c34 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0706035f-9cec-459b-b598-1ee8165a283b_calibrate_1039672_13.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0706035f-9cec-459b-b598-1ee8165a283b_calibrate_1039672_13.stderr parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4120e8fe1d7f81969a967f9959e4ac4 with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash a9851dbf06a4ef8d5501ff2b595c5ac6 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ed846e42-e7dc-490b-bb5b-3d2c883938aa_calibrate_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ed846e42-e7dc-490b-bb5b-3d2c883938aa_calibrate_1039770_40.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a92bc859-02c0-4859-9f59-0b6e551f7bbe_characterizeImage_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a92bc859-02c0-4859-9f59-0b6e551f7bbe_characterizeImage_1039770_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:18 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d80a5dfe76249cc227aaa09c62255ad2 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash d3f0b3f9a439d13565bf0e0c04207fc8 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/332f0ce0-ccf4-4622-ae70-fb14df63aec2_calibrate_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/332f0ce0-ccf4-4622-ae70-fb14df63aec2_calibrate_1039770_25.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ebbd6cd3-b82b-4d6c-a83a-a43c370a9ef9_characterizeImage_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ebbd6cd3-b82b-4d6c-a83a-a43c370a9ef9_characterizeImage_1039770_25.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4843352daa91e237dde976baf232ce with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash fe068d655f9e2b9c548c44067eb9d052 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/10b0040f-2136-495f-a52d-baa4f483b252_calibrate_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/10b0040f-2136-495f-a52d-baa4f483b252_calibrate_1039768_24.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/202cc26a-8cea-4094-b2e9-57cfbaeb25e0_characterizeImage_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/202cc26a-8cea-4094-b2e9-57cfbaeb25e0_characterizeImage_1039768_24.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bedf7e7161dc1b92f830b308e7a27177 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash e7c3bf731ce20a8209331887f20c85e6 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/467834c5-f5cf-46b6-950a-8f97644d5ce6_calibrate_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/467834c5-f5cf-46b6-950a-8f97644d5ce6_calibrate_1039674_18.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/63af07f6-5bd0-4cb1-a6ed-4f0d48d2a032_characterizeImage_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/63af07f6-5bd0-4cb1-a6ed-4f0d48d2a032_characterizeImage_1039674_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:18 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c883b7788613707bd37c85f6e7ed588 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash ba35adc8b0f8b40471cd3a987ebe842f parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5388ac1e-3a49-4e01-b222-a3ab5c526139_calibrate_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5388ac1e-3a49-4e01-b222-a3ab5c526139_calibrate_1039769_13.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d02b06b2-5d62-4440-a94d-43eb6eeefe6c_characterizeImage_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d02b06b2-5d62-4440-a94d-43eb6eeefe6c_characterizeImage_1039769_13.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82da744ae84424c2420986ea505aa702 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 5593d2a9704ba6d43a60052d292a8286 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/fa3eb287-b3e0-43f6-871b-7419375f1d5b_calibrate_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/fa3eb287-b3e0-43f6-871b-7419375f1d5b_calibrate_1039673_10.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/30aea6b1-daea-498f-a198-66c20a56095e_characterizeImage_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/30aea6b1-daea-498f-a198-66c20a56095e_characterizeImage_1039673_10.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fdddaa6698d7e9cdf0c93f4edc11f58 with result from task 508 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:19 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 2e172475fa90a55e7fcf83c415c3a69d parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3ba200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/a26c7cd2-1f5c-4247-8584-1f0a2b7baf28_calibrate_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/a26c7cd2-1f5c-4247-8584-1f0a2b7baf28_calibrate_1039769_5.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/aa32f058-1b8b-4d76-b4bc-5e11d1283a51_characterizeImage_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/aa32f058-1b8b-4d76-b4bc-5e11d1283a51_characterizeImage_1039769_5.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dd951d27e2ed218605b5d031df3d424 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/95bc0b02-0dd8-4148-b199-69dfd292720b_calibrate_1039770_36.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/95bc0b02-0dd8-4148-b199-69dfd292720b_calibrate_1039770_36.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5124bc0b1ce59bf73a519b271dc17226 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 38249480e96d6722c7f17e1ee027aabb parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3800e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e18548a2-6c54-4ebc-9a8b-d721fd64d9e5_calibrate_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e18548a2-6c54-4ebc-9a8b-d721fd64d9e5_calibrate_1039769_10.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/9f1255a8-a7a7-4762-a226-af6343e1f403_characterizeImage_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/9f1255a8-a7a7-4762-a226-af6343e1f403_characterizeImage_1039769_10.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa764272a90a443094417bea87acad5 with result from task 185 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:19 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/94c7da42-339a-4fb1-9147-a6739ab75526_calibrate_1039674_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/94c7da42-339a-4fb1-9147-a6739ab75526_calibrate_1039674_48.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d5c48157d93f147eaddc2c0aecdccb6 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 3bd984a8e816a0c0255400560fba3903 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4d2ba30f-38d2-47c9-8cf8-5eccc5b60afa_calibrate_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4d2ba30f-38d2-47c9-8cf8-5eccc5b60afa_calibrate_1039768_10.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e6c629d9-750a-43da-a30b-cd33e4b1db96_characterizeImage_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e6c629d9-750a-43da-a30b-cd33e4b1db96_characterizeImage_1039768_10.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf25929da1c7849943c33f5ea2d235a3 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 06aaa1cf442b4d7b3824e3cfc663f783 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/903038a0-9170-4798-8124-c06246dd3339_calibrate_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/903038a0-9170-4798-8124-c06246dd3339_calibrate_1039770_7.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cceaf9b8-16f8-4fd0-beb7-5c3e6c70891d_characterizeImage_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cceaf9b8-16f8-4fd0-beb7-5c3e6c70891d_characterizeImage_1039770_7.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f614f8e3f40bb4f5d472213410001da8 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6c3fd182-e139-4cfd-9725-ef75181de375_calibrate_1039768_32.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6c3fd182-e139-4cfd-9725-ef75181de375_calibrate_1039768_32.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 349df24d3ae3b408a6df318dfbe53447 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6a2bdf4e-09b2-4532-847b-3a3277246cc6_calibrate_1039674_12.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6a2bdf4e-09b2-4532-847b-3a3277246cc6_calibrate_1039674_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:20 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35fa94f31c3886d866bd348c6df698ee with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e7889509-b738-4b6c-95e3-a87476f51637_calibrate_1039769_28.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e7889509-b738-4b6c-95e3-a87476f51637_calibrate_1039769_28.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5437543d2d9dc87bdff07f7100456a27 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14fe8bf9-30c5-4ca9-ad29-b5b2f4f5b622_calibrate_1039674_6.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14fe8bf9-30c5-4ca9-ad29-b5b2f4f5b622_calibrate_1039674_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:21 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4891e7773050a3e31dbd7eceebeaf44 with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4699c771-a670-46c8-acfd-797fe80208bd_calibrate_1039770_18.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4699c771-a670-46c8-acfd-797fe80208bd_calibrate_1039770_18.stderr parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90508cbcf6904e8924ee852b8be721be with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 3cc996a245b26195c9608711fd1bf8d1 parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3d373a3c-3bb6-48f8-9660-8c49db0eb23a_calibrate_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3d373a3c-3bb6-48f8-9660-8c49db0eb23a_calibrate_1039768_21.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/35f3e69c-ea30-493d-a9c4-2adee87c86bc_characterizeImage_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/35f3e69c-ea30-493d-a9c4-2adee87c86bc_characterizeImage_1039768_21.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237c062ab715347acd8250a01ff39689 with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 8253ef868f3bb048d43c7f3a34bc342b parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/967e1ba4-9e32-47bc-b82b-2aba5fcd566a_calibrate_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/967e1ba4-9e32-47bc-b82b-2aba5fcd566a_calibrate_1039673_62.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5004d6f8-a1c9-4794-94dd-c78f3e29d987_characterizeImage_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5004d6f8-a1c9-4794-94dd-c78f3e29d987_characterizeImage_1039673_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:21 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:22 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5333ed3f908e7e69353584e19697ebeb with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1ee9b0e2-dfde-4513-9e87-0b9cfc3998fc_calibrate_1039770_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1ee9b0e2-dfde-4513-9e87-0b9cfc3998fc_calibrate_1039770_1.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c340009f4938356cf90437d9ed672798 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash aab5f6876c7aac4f6b5b5c451694354e parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4e8fb25c-c694-4360-87ef-82a034236537_calibrate_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4e8fb25c-c694-4360-87ef-82a034236537_calibrate_1039770_53.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ec4b12fc-4602-4daf-aa39-628192b0a4e5_characterizeImage_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ec4b12fc-4602-4daf-aa39-628192b0a4e5_characterizeImage_1039770_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:22 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a3a6937bbcd0966dc1590b9c77cb28e with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 0c3f92cb6d754589efa569e7c64178e1 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/79e98967-02dc-49b7-a09f-11a10a987ab6_calibrate_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/79e98967-02dc-49b7-a09f-11a10a987ab6_calibrate_1039769_54.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/da4f4f2a-9695-4500-b696-34239efd0398_characterizeImage_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/da4f4f2a-9695-4500-b696-34239efd0398_characterizeImage_1039769_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:22 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec58af8d6063cec4f9ef5276679b23e8 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash e67d8fc8d3f5fcca339bfbce116554d0 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/95406503-fd2a-45a6-a443-2a2261048888_calibrate_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/95406503-fd2a-45a6-a443-2a2261048888_calibrate_1039768_4.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bfbce4d5-c145-4810-907d-61a9971ecbe3_characterizeImage_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/bfbce4d5-c145-4810-907d-61a9971ecbe3_characterizeImage_1039768_4.stderr parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bfd13d2f9934e32583f9265c267c721 with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash 9225cd40eee332e8d0576a67d066ca45 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/99208013-9889-4f30-bc27-a8353c5dd353_calibrate_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/99208013-9889-4f30-bc27-a8353c5dd353_calibrate_1039674_44.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/628ac53d-89cc-4daf-8589-f16e368d2df0_characterizeImage_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/628ac53d-89cc-4daf-8589-f16e368d2df0_characterizeImage_1039674_44.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83b6e642059045b600582fea958eb446 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 9f0b090af94d3af2adee5e8d4d4133e3 parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f102153c-9c4b-4888-99b7-5fe2212f509c_calibrate_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f102153c-9c4b-4888-99b7-5fe2212f509c_calibrate_1039768_49.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8419c5ca-655c-4f2e-9276-d1b509b7d5df_characterizeImage_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8419c5ca-655c-4f2e-9276-d1b509b7d5df_characterizeImage_1039768_49.stderr parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 874f86bdc1ca4b64935cd575138435ef with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash cc4583b7b4acc46224fcd7eb1726dbc3 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0da58550-e457-4f5a-a5f4-fd7798ed58b0_calibrate_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0da58550-e457-4f5a-a5f4-fd7798ed58b0_calibrate_1039769_12.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/129b02d3-f9e0-481b-9963-17905ecffe12_characterizeImage_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/129b02d3-f9e0-481b-9963-17905ecffe12_characterizeImage_1039769_12.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75be264c35a0bc598485ce02d0027513 with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 11d981f93962e34598f28cf8d4d50e15 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e278680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c7130d0-d3ba-428f-9899-602f23b0d657_calibrate_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c7130d0-d3ba-428f-9899-602f23b0d657_calibrate_1039673_9.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/27006bec-3b06-4568-9a5b-bcf39a0827a1_characterizeImage_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/27006bec-3b06-4568-9a5b-bcf39a0827a1_characterizeImage_1039673_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:23 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b20e914281c9c562d25fa5cb9fc3655 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash c8b3c21d1e2c65f999732c316148a61d parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/43d5e0c2-4cc2-484e-a05d-c7949942dac1_calibrate_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/43d5e0c2-4cc2-484e-a05d-c7949942dac1_calibrate_1039770_23.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a2dd0dd9-ef9b-4590-a907-a97ff58a9abc_characterizeImage_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a2dd0dd9-ef9b-4590-a907-a97ff58a9abc_characterizeImage_1039770_23.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e86f4bbb1b01a8c8a81c8c73d4efe1c with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 9799356fafc093ec31dee0c7174c42ea parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5d136ede-915d-4998-a5ee-a229b6e9741a_calibrate_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5d136ede-915d-4998-a5ee-a229b6e9741a_calibrate_1039673_31.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bb00c70a-1686-4414-aa25-73629b248351_characterizeImage_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/bb00c70a-1686-4414-aa25-73629b248351_characterizeImage_1039673_31.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b10ff0d0839d12ba7652921f17b7bba3 with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash e28e707529d2b8d824ec61ff815c1c48 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bd22e264-964e-4b50-96e5-d6254a7af605_calibrate_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bd22e264-964e-4b50-96e5-d6254a7af605_calibrate_1039769_52.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0222b0b6-d361-4995-8fbe-2a03a0bdad01_characterizeImage_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/0222b0b6-d361-4995-8fbe-2a03a0bdad01_characterizeImage_1039769_52.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3822e4c6f19ff0c234e0f9f7d03a5ca8 with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash fa623ccd543a54df48e59fe41d783a2f parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/61f10922-a549-43b4-a177-00c56e87a426_calibrate_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/61f10922-a549-43b4-a177-00c56e87a426_calibrate_1039769_42.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/47b164c6-9fa8-4a82-b960-6ce3780abb17_characterizeImage_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/47b164c6-9fa8-4a82-b960-6ce3780abb17_characterizeImage_1039769_42.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdc8528bd4c25b57bf78a50db2b4f489 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash be4b2be24b1baea46ac27100ed989fee parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e214cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f5550704-a5c9-4959-ac9f-559210c13290_calibrate_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f5550704-a5c9-4959-ac9f-559210c13290_calibrate_1039768_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e8daf669-4aef-4285-b55a-4f63d8ecf040_characterizeImage_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e8daf669-4aef-4285-b55a-4f63d8ecf040_characterizeImage_1039768_1.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87af97eb0eb260a40f1f83c66d78de2d with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 15266c5be337cc2589417cd55d81ac5b parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/9e397aa9-1fca-4efd-a102-7e4bb2637011_calibrate_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/9e397aa9-1fca-4efd-a102-7e4bb2637011_calibrate_1039769_4.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3ffaa2dd-2dab-44b7-98ed-4c54b8813270_characterizeImage_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3ffaa2dd-2dab-44b7-98ed-4c54b8813270_characterizeImage_1039769_4.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80d5136c3b6ba9cf63c7e2fa4b13e73a with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 78d7646bdd92ec874312418b4dd897dc parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b5e458c-7f47-45b9-b3a6-5cb642d2f19e_calibrate_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b5e458c-7f47-45b9-b3a6-5cb642d2f19e_calibrate_1039673_37.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a609c444-d528-4554-821e-41db6efe59d7_characterizeImage_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a609c444-d528-4554-821e-41db6efe59d7_characterizeImage_1039673_37.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27918c062798519134fcd6a68c33bcc1 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash f9c8aaed0d9b6feccee75ad2451ad580 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/450010c5-5b5f-417d-93ae-c916b151be8e_calibrate_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/450010c5-5b5f-417d-93ae-c916b151be8e_calibrate_1039768_12.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/453e38c0-0bf6-429d-a902-19ebb161416c_characterizeImage_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/453e38c0-0bf6-429d-a902-19ebb161416c_characterizeImage_1039768_12.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6620c2b3bcfa1985ad0a9061f692c57 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 16b1d0aeb5a704d990bbf9b58a7f5f53 parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/290e4bed-d6c6-49e5-a515-7e444fdfc2d1_calibrate_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/290e4bed-d6c6-49e5-a515-7e444fdfc2d1_calibrate_1039672_12.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f6946435-35e3-4f3b-846a-6750f7009f75_characterizeImage_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/f6946435-35e3-4f3b-846a-6750f7009f75_characterizeImage_1039672_12.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56a2fc500599a5f8ec64ac43c584dea5 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash ee3019ae2f67fb7f709603cf39034c9e parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1712f189-5092-4bf8-a31c-d5d4de533ca7_calibrate_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1712f189-5092-4bf8-a31c-d5d4de533ca7_calibrate_1039673_26.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c67850ad-5d62-45e9-a8c0-866b52333020_characterizeImage_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c67850ad-5d62-45e9-a8c0-866b52333020_characterizeImage_1039673_26.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49fc0a323b9f85efc2965b7a062877ea with result from task 247 parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:24 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash c56c8e2c8819e98753711ac650ad62e7 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ad152488-4319-485b-99b6-3b3750a0f1b8_calibrate_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ad152488-4319-485b-99b6-3b3750a0f1b8_calibrate_1039770_34.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6f7fcdec-2285-48c8-9816-80787ae1fbce_characterizeImage_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6f7fcdec-2285-48c8-9816-80787ae1fbce_characterizeImage_1039770_34.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd8273b84c60426cdac2d67a8db9f75f with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/e00a0c82-fa76-4e82-96be-093bfbb912b4_calibrate_1039770_21.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/e00a0c82-fa76-4e82-96be-093bfbb912b4_calibrate_1039770_21.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f3d873df612e3ca3454eaf6c475cde4 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 96a14849c38b076750d4a412921df234 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5eace840-91bb-419a-86a5-f9e3efe2068a_calibrate_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5eace840-91bb-419a-86a5-f9e3efe2068a_calibrate_1039768_62.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d5b29802-39f6-4953-9c3c-9cbf51cb3b96_characterizeImage_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d5b29802-39f6-4953-9c3c-9cbf51cb3b96_characterizeImage_1039768_62.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d17d8fcfb3163267d007bc502cde381 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0b2ae7ec-0ac6-4c56-be2d-5b6c9f3b277c_calibrate_1039674_28.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0b2ae7ec-0ac6-4c56-be2d-5b6c9f3b277c_calibrate_1039674_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cca485f1729694a67bf511d3338637ad with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/113e5afe-9857-4b3b-9c1f-c5109921e414_calibrate_1039672_52.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/113e5afe-9857-4b3b-9c1f-c5109921e414_calibrate_1039672_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:25 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccb0208067ffae947d8d295157999a3a with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash e7320a61de770f60874a854058ef0a3d parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f62e036a-d033-456a-bca0-420b61e2ad83_calibrate_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f62e036a-d033-456a-bca0-420b61e2ad83_calibrate_1039672_9.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e4e3062-a45a-442f-917e-69d2dcd4f6d9_characterizeImage_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e4e3062-a45a-442f-917e-69d2dcd4f6d9_characterizeImage_1039672_9.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05d35316c02c6f7bb6ef1c18e1c16cdf with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash ae019ab654cdfb247617185ed9940579 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/30c07768-e7f2-45bf-9df6-2ad679ec70c1_calibrate_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/30c07768-e7f2-45bf-9df6-2ad679ec70c1_calibrate_1039674_16.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9b088661-0f38-4631-9ede-92ab43da17bd_characterizeImage_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/9b088661-0f38-4631-9ede-92ab43da17bd_characterizeImage_1039674_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 849e4da7b13933bda45c77873e25be68 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash c3e51f4a8a6bf2dd41c6201d07783960 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14943ec1-4068-47d5-8c8a-055536328d0f_calibrate_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14943ec1-4068-47d5-8c8a-055536328d0f_calibrate_1039674_5.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0214e764-064b-4482-a3bf-651280fd530a_characterizeImage_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/0214e764-064b-4482-a3bf-651280fd530a_characterizeImage_1039674_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8938c165f04a041ed31c3d701bb5811 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash d0a4bf7ba69dbd718666355f2fdfc4e1 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e27ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3a3b223d-76b9-421b-b5ca-564c3221516a_calibrate_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3a3b223d-76b9-421b-b5ca-564c3221516a_calibrate_1039769_8.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d3f2d61-5942-4cd7-bac4-a317ac2fea6b_characterizeImage_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d3f2d61-5942-4cd7-bac4-a317ac2fea6b_characterizeImage_1039769_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:26 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:27 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98a1b0f44ae04ba96a6bfd8f73516ea1 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 4238661fee35ce1943733f42806ed7e1 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d493e19-afc0-49bb-a624-af8d0b1ac5af_calibrate_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d493e19-afc0-49bb-a624-af8d0b1ac5af_calibrate_1039672_55.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7bc1b523-797e-4e90-9360-a3cbf17e1098_characterizeImage_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7bc1b523-797e-4e90-9360-a3cbf17e1098_characterizeImage_1039672_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:27 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e776f7b87610cae8a25b36d64609cff0 with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 1e7a38408fbf532a3a245a362a2a3f5d parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea4d2830-a950-43e0-ba26-82396dafd714_calibrate_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea4d2830-a950-43e0-ba26-82396dafd714_calibrate_1039769_6.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/63fe4c09-0eae-4edf-a839-284b8d2407a4_characterizeImage_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/63fe4c09-0eae-4edf-a839-284b8d2407a4_characterizeImage_1039769_6.stderr parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537a3dc37841327cb27963f051a2ad81 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6cad7ec2-fcaf-4caa-bfbe-fe8a43c356d0_calibrate_1039674_24.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6cad7ec2-fcaf-4caa-bfbe-fe8a43c356d0_calibrate_1039674_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:27 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac3b1af1a69d757e79dbc95eb85a594 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 63b0deaa0f560e0b145827fd1e48e7c7 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c5a3fcea-2052-4d64-9894-e181636af026_calibrate_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c5a3fcea-2052-4d64-9894-e181636af026_calibrate_1039769_25.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b5e0ad22-0273-4a03-b5b1-a8cd7bdb8dab_characterizeImage_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b5e0ad22-0273-4a03-b5b1-a8cd7bdb8dab_characterizeImage_1039769_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:27 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:27 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 162853302feb852a8ac8c011c4f12748 with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash f20c4749eaafda22c9658df4cc6d695d parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/20272cbb-80a3-414d-91af-efcffe25a1ae_calibrate_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/20272cbb-80a3-414d-91af-efcffe25a1ae_calibrate_1039672_31.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3b0c1ae3-9882-47f0-b3d1-47e5023d126d_characterizeImage_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3b0c1ae3-9882-47f0-b3d1-47e5023d126d_characterizeImage_1039672_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:28 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e1b549244db664f7e78733e22142268 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash e2710caab0ed11e221ee40a686ab2eb3 parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b68e3803-4ce2-4e6c-a25c-ee495a2a5a2c_calibrate_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b68e3803-4ce2-4e6c-a25c-ee495a2a5a2c_calibrate_1039769_51.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d9ac676-b90a-4e07-a694-b721eacb8099_characterizeImage_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d9ac676-b90a-4e07-a694-b721eacb8099_characterizeImage_1039769_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:28 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d2309218952b47cc2d696797c19fa62 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash e1362c9eff27888318228eef71116de3 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2feac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c03eea5f-df62-437f-be2d-8e9b5e268a31_calibrate_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c03eea5f-df62-437f-be2d-8e9b5e268a31_calibrate_1039769_20.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c4129ac7-06c9-422c-84a4-1059f541ab3a_characterizeImage_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c4129ac7-06c9-422c-84a4-1059f541ab3a_characterizeImage_1039769_20.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry beb19f745a68f2e9cdbf2bb2c168a6ce with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 1149a0841d7d8030b948a168851480fe parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d2255967-b238-4c6c-be0f-418688ba182e_calibrate_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d2255967-b238-4c6c-be0f-418688ba182e_calibrate_1039674_19.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/49d5f573-facb-4505-aba1-bc5f4fdd8573_characterizeImage_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/49d5f573-facb-4505-aba1-bc5f4fdd8573_characterizeImage_1039674_19.stderr parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45df89ed83d60cca918ca73c8127fab1 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash 894133cd79eb0823190d5bb8416a1bf4 parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7887f4c1-5da2-4995-8a7d-a6bd07d56793_calibrate_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7887f4c1-5da2-4995-8a7d-a6bd07d56793_calibrate_1039673_47.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/aec25512-d293-4b7c-9185-56be329015c7_characterizeImage_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/aec25512-d293-4b7c-9185-56be329015c7_characterizeImage_1039673_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:28 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd8653e2f44c3783ecdbf6520610d2d0 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/71b9c34f-4ca7-4485-975b-f7925947a650_calibrate_1039672_17.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/71b9c34f-4ca7-4485-975b-f7925947a650_calibrate_1039672_17.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b027d4ad1022d0de89dda95f4b9625c with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 066734d917e661afdb0d2a0db08a6beb parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/568d7614-1d4d-4981-b2ec-248e6c41f584_calibrate_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/568d7614-1d4d-4981-b2ec-248e6c41f584_calibrate_1039672_60.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/24c8414e-8979-4da1-8efb-528def55e7ec_characterizeImage_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/24c8414e-8979-4da1-8efb-528def55e7ec_characterizeImage_1039672_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:29 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d03f7f1fed48a5ef6fa2b786ac4ba72c with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash bd9e98f3ff236653477a4dc15a5510e7 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/27142b82-9401-46af-b8db-f3d4bd1d9322_calibrate_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/27142b82-9401-46af-b8db-f3d4bd1d9322_calibrate_1039769_19.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4cf765bf-6964-4cfb-b18d-a9fd9003350f_characterizeImage_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4cf765bf-6964-4cfb-b18d-a9fd9003350f_characterizeImage_1039769_19.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfb4d50c66f12e7c7d603e84d6a4a076 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f9f7e911-8a0b-402a-a3a2-8fe45139cda8_calibrate_1039674_22.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f9f7e911-8a0b-402a-a3a2-8fe45139cda8_calibrate_1039674_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:29 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:30 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91656d3fbe253cf9b8bfc431a7abee63 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash fc6e9e432a5117d68178010445922005 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/abce862a-9a82-4af2-be99-ea87a2f25898_calibrate_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/abce862a-9a82-4af2-be99-ea87a2f25898_calibrate_1039769_41.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b33c9068-d7db-4615-85eb-2d96888a07c8_characterizeImage_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/b33c9068-d7db-4615-85eb-2d96888a07c8_characterizeImage_1039769_41.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf99c7630809c122968434f7af3aa388 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3ea7a349-dcbf-47dc-b15f-7e08f4d7f1ad_calibrate_1039769_46.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3ea7a349-dcbf-47dc-b15f-7e08f4d7f1ad_calibrate_1039769_46.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1cd223b7b43992a4e350b55be440fe6 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/c49847c2-ea14-4dc6-887a-ad301707365b_calibrate_1039768_3.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/c49847c2-ea14-4dc6-887a-ad301707365b_calibrate_1039768_3.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bca534d94a3611ae378e185604473b94 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 574b15a420b7463725ea05060925dcd6 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/dc885268-1a4c-4f56-a9f0-7a01253c6d2d_calibrate_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/dc885268-1a4c-4f56-a9f0-7a01253c6d2d_calibrate_1039672_23.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5510bb8-bea2-4be8-b3d2-04aafdfc1e05_characterizeImage_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/c5510bb8-bea2-4be8-b3d2-04aafdfc1e05_characterizeImage_1039672_23.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ac77416945013922440ff9ce7e9351e with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash e3e06d6598ea3ecb0196b83e8441dfe5 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fe160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0d54a077-66a5-477c-89f0-d1f2a4ab3250_calibrate_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0d54a077-66a5-477c-89f0-d1f2a4ab3250_calibrate_1039672_37.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9ea67fc6-6984-4963-a113-a5e9c5009ff4_characterizeImage_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/9ea67fc6-6984-4963-a113-a5e9c5009ff4_characterizeImage_1039672_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:31 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471a6aba10cb913e9f4913061b48a067 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 31df165789bc7481827cc383b376a3f2 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c62b57a7-1561-4226-beb5-769588482f24_calibrate_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c62b57a7-1561-4226-beb5-769588482f24_calibrate_1039770_60.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2832a6ed-fc93-45a5-8f4c-7b0f7ce8aa11_characterizeImage_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2832a6ed-fc93-45a5-8f4c-7b0f7ce8aa11_characterizeImage_1039770_60.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a3cba24a6b64e802eaf1810fbc79c6e with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f57ba267-e7a0-429d-a20a-604c80452296_calibrate_1039673_24.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/f57ba267-e7a0-429d-a20a-604c80452296_calibrate_1039673_24.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5d77854817e83d76b8c7abbc6d22cd2 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 9c04c1dfaad5f662e3f72f53f24d2c28 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e784b519-2f79-4707-9c69-a3b3b9376001_calibrate_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e784b519-2f79-4707-9c69-a3b3b9376001_calibrate_1039673_54.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/73938340-9778-42a6-98d6-97646ebce3c2_characterizeImage_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/73938340-9778-42a6-98d6-97646ebce3c2_characterizeImage_1039673_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:31 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 196d2b1967c4a75bb922ac89241f6e4f with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash a43bac5b7ac56f94495206db7956043c parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cd28e715-9072-4a81-8ee6-e03541c4f204_calibrate_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cd28e715-9072-4a81-8ee6-e03541c4f204_calibrate_1039769_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a38cb0-c317-4ef4-af17-9a8710af5cf4_characterizeImage_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/49a38cb0-c317-4ef4-af17-9a8710af5cf4_characterizeImage_1039769_17.stderr parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e20231e8c6a2dce3da37b256c270c79 with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash e47f2be449cd68f6ff044d50a7db4a07 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/2c098d4f-2f6f-49f7-a6ec-875dae1d72b6_calibrate_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/2c098d4f-2f6f-49f7-a6ec-875dae1d72b6_calibrate_1039672_44.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/68b0036a-64b4-4fbd-aab0-1019e0d65edd_characterizeImage_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/68b0036a-64b4-4fbd-aab0-1019e0d65edd_characterizeImage_1039672_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:32 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 019d3df643bba435ca97c3da4a58d1dc with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash e6432376280d99f79bc09348240d5156 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11ec19b4-e9b4-43e4-a1d0-6f58307fb089_calibrate_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11ec19b4-e9b4-43e4-a1d0-6f58307fb089_calibrate_1039770_24.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0cd0be50-fa26-41e4-9fbf-41affb874cca_characterizeImage_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/0cd0be50-fa26-41e4-9fbf-41affb874cca_characterizeImage_1039770_24.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 744db058e4ad2f6d4ca1e99c305bd84a with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 85244575c0f5c474b4e1f98f77cb1ca9 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/95553e53-03e8-42e5-bef1-34a241fa103c_calibrate_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/95553e53-03e8-42e5-bef1-34a241fa103c_calibrate_1039672_26.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1b1c58bb-1a59-4e9d-96d5-be4b4556ba3e_characterizeImage_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/1b1c58bb-1a59-4e9d-96d5-be4b4556ba3e_characterizeImage_1039672_26.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5987ebb543f7200a826f169fb817c8 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash ada818e16c768e7619ba989c7fcab709 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/04d3d496-ba2c-4655-bb44-474dd663d56d_calibrate_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/04d3d496-ba2c-4655-bb44-474dd663d56d_calibrate_1039673_34.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2ef04e56-5442-4573-bd79-18c41c0ad30e_characterizeImage_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/2ef04e56-5442-4573-bd79-18c41c0ad30e_characterizeImage_1039673_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:33 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65caedcbd8dbf64d1a9cd38c6e25b820 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash ce76ecfa5a04ce5422c0e9dea11f7985 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4c72a321-0c09-42fe-8024-52592f3c9ded_calibrate_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4c72a321-0c09-42fe-8024-52592f3c9ded_calibrate_1039770_51.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/331767c9-6761-40bd-85fe-c8f7999d593d_characterizeImage_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/331767c9-6761-40bd-85fe-c8f7999d593d_characterizeImage_1039770_51.stderr parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e65c8a1b3074993038fa8f73e36603e with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 26fb4febef2d147441f40c984f4a9bda parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d4d51fe6-d791-43d9-acc1-5dce218abc03_calibrate_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d4d51fe6-d791-43d9-acc1-5dce218abc03_calibrate_1039770_27.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/31263d4b-4d2e-4aef-9994-866cf17ce023_characterizeImage_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/31263d4b-4d2e-4aef-9994-866cf17ce023_characterizeImage_1039770_27.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924a5786438e6e82b34c312a5ea094db with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 100973a542f39532f049a495ace26b2d parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b3358c93-e8a2-4e4e-8fcc-adda62d2cab9_calibrate_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b3358c93-e8a2-4e4e-8fcc-adda62d2cab9_calibrate_1039770_49.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/aaf10ef4-e8ca-47d1-a0ec-fd583db8a7f8_characterizeImage_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/aaf10ef4-e8ca-47d1-a0ec-fd583db8a7f8_characterizeImage_1039770_49.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df5b3d194f81495ccab8336a243c5f0 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0c47d7d1-2925-493a-a40f-f2736324dfad_calibrate_1039769_24.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0c47d7d1-2925-493a-a40f-f2736324dfad_calibrate_1039769_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:33 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc025759fea03fd444676502159f844d with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/dc22b7c7-a23f-4975-ba78-df381be9cdc3_calibrate_1039769_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/dc22b7c7-a23f-4975-ba78-df381be9cdc3_calibrate_1039769_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:33 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65b11b1bb58828c877b4b6d64d2aad13 with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash a151b3fe203e3836068e33e24aa19ef7 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/eb11b2ac-498b-4281-9349-d80f52d2ba6e_calibrate_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/eb11b2ac-498b-4281-9349-d80f52d2ba6e_calibrate_1039770_19.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9f2c1ef7-45a7-45c7-ab14-5a253453256a_characterizeImage_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/9f2c1ef7-45a7-45c7-ab14-5a253453256a_characterizeImage_1039770_19.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 922d150efa60ccdf5b2217c2f192850d with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fa738dbc-51a8-4765-a0fa-dbe1fa803ab7_calibrate_1039672_7.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fa738dbc-51a8-4765-a0fa-dbe1fa803ab7_calibrate_1039672_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:34 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b38751682943fbd25eb561c8bfebdc99 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash a8670d16960f2cea0cbf1a90d9e15e0c parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e382d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/23e0a77b-6984-4016-9b54-ffbdb00a7e42_calibrate_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/23e0a77b-6984-4016-9b54-ffbdb00a7e42_calibrate_1039673_16.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/270c34ea-32b5-4166-b9de-2a07547121a1_characterizeImage_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/270c34ea-32b5-4166-b9de-2a07547121a1_characterizeImage_1039673_16.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862e07e53b6c1714f4837a50bced6dfe with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dbefda3a-d4bd-4563-8e40-c25d729b44f1_calibrate_1039673_35.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dbefda3a-d4bd-4563-8e40-c25d729b44f1_calibrate_1039673_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:34 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bd1d7ae9cf184f629a65c185bd20049 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d6c78836-12c9-497c-8307-6bcfc13779cb_calibrate_1039770_20.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d6c78836-12c9-497c-8307-6bcfc13779cb_calibrate_1039770_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:35 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9548f222c8d1099a402ebcf7cf6e27f8 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 5c5efb7a8768fc039d57646f64021c2c parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e074cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1f7440c8-aba1-4602-bba7-b7ca4863a7cf_calibrate_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1f7440c8-aba1-4602-bba7-b7ca4863a7cf_calibrate_1039673_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c49f471-6b63-4656-a7ac-1ed5f4728afe_characterizeImage_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/5c49f471-6b63-4656-a7ac-1ed5f4728afe_characterizeImage_1039673_3.stderr parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cf976debff7e7120b3f76e357c80b04 with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d691dec1-4a85-4d97-95b1-c37f7740404d_calibrate_1039674_3.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d691dec1-4a85-4d97-95b1-c37f7740404d_calibrate_1039674_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:35 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34feebcea330997127a8d0b838bf6c16 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 3da3053a83fd2e587e0a1443ef55e343 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/382fabb3-2993-45a0-b00c-52d825524327_calibrate_1039674_50.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/382fabb3-2993-45a0-b00c-52d825524327_calibrate_1039674_50.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e26b1815-91bc-44ad-893a-9261a7f8c04a_characterizeImage_1039674_50.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/e26b1815-91bc-44ad-893a-9261a7f8c04a_characterizeImage_1039674_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:36 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffd4f22abf60ebf10ad3e9ebc8f11e1 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0357ec04-a2b6-4d5b-bb7d-81fc7d84f2a3_calibrate_1039769_14.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0357ec04-a2b6-4d5b-bb7d-81fc7d84f2a3_calibrate_1039769_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:36 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08123c6bb892c2ad3b53ff99c5fa6e55 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/acf7e7fe-4f50-49c4-a6fd-0cf9e67a55e2_calibrate_1039672_11.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/acf7e7fe-4f50-49c4-a6fd-0cf9e67a55e2_calibrate_1039672_11.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66f549662ed00b31c73124982a1c337c with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ecb549db-3709-460a-a3db-049dc18a5369_calibrate_1039768_55.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ecb549db-3709-460a-a3db-049dc18a5369_calibrate_1039768_55.stderr parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc2a6615dfd8e4aa6297a8ed6fd2eaa7 with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3872e470-b259-4bc6-8490-94bd8a255f59_calibrate_1039770_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3872e470-b259-4bc6-8490-94bd8a255f59_calibrate_1039770_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:37 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ccf724c0f15cf4754b62ba342c84a5c with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/312a9cf8-b993-4a6d-9dc5-5a6a02233509_calibrate_1039674_49.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/312a9cf8-b993-4a6d-9dc5-5a6a02233509_calibrate_1039674_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:37 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30fccf15ba7c9c67e8b0d57f8bfa80ba with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 5d2848c2b10d129e6f94b89071888144 parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7ae14e33-a555-4d50-a896-8d0453d5a612_calibrate_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7ae14e33-a555-4d50-a896-8d0453d5a612_calibrate_1039769_47.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/cb8af2e1-d841-4ff6-9a11-81c40c98c97a_characterizeImage_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/cb8af2e1-d841-4ff6-9a11-81c40c98c97a_characterizeImage_1039769_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:37 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63914a521ff87c9f032f2b0b3641f4ff with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 63fa7178c7052eed8a761dc28f6cb273 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1ebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f551c325-7ef2-482f-9ea0-1d419a008276_calibrate_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f551c325-7ef2-482f-9ea0-1d419a008276_calibrate_1039672_41.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/75628134-4981-4882-b3b8-2cbffea08018_characterizeImage_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/75628134-4981-4882-b3b8-2cbffea08018_characterizeImage_1039672_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:37 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5e969191a5fd0d60875abb133cacde7 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5ece5783-2e17-4bcd-9693-cca992d7a776_calibrate_1039770_33.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5ece5783-2e17-4bcd-9693-cca992d7a776_calibrate_1039770_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:37 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6a5cbd37dc5785ebdeed396ed29d31 with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 61b6552bda0dca02d330c3e1c8ee4c66 parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e0749a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/48c20a88-3966-478a-8460-44c5474d85de_calibrate_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/48c20a88-3966-478a-8460-44c5474d85de_calibrate_1039770_31.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6411996d-0584-42d1-b63c-fa76777072e6_characterizeImage_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/6411996d-0584-42d1-b63c-fa76777072e6_characterizeImage_1039770_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:38 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c69238b93bb10415ab85be27b164f8b with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b7b4a2bb-5e83-49ed-a684-4faefb869531_calibrate_1039770_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b7b4a2bb-5e83-49ed-a684-4faefb869531_calibrate_1039770_22.stderr parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c15992dd5ea8834d5e2c28d05c34bf with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 653b3c81c9db2cf20dded2a804cdbbed parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3eef43cc-bc8f-4d54-a703-a4f93f5c6ebb_calibrate_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3eef43cc-bc8f-4d54-a703-a4f93f5c6ebb_calibrate_1039770_39.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/699358c1-8228-4c6a-84ee-19cd5a779c62_characterizeImage_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/699358c1-8228-4c6a-84ee-19cd5a779c62_characterizeImage_1039770_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 546ee9715678a632be2cc79306176626 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash ee270d1ca0eaf01c2978b9f39eecabae parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fe6320c4-bb23-4f3c-af7b-db0e30da08eb_calibrate_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fe6320c4-bb23-4f3c-af7b-db0e30da08eb_calibrate_1039768_29.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e889cafc-0a6a-438e-ac45-43cb7f5775a3_characterizeImage_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e889cafc-0a6a-438e-ac45-43cb7f5775a3_characterizeImage_1039768_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ba3bedf4104245db52fd58b1ddacc9 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1436f68e-9d4c-44b8-b255-1c34210d554b_calibrate_1039770_10.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/1436f68e-9d4c-44b8-b255-1c34210d554b_calibrate_1039770_10.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b9da68ecabd1c79794d0ba6dcfd9e69 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 92f6449cdc90c0f65de1c3dc2875e151 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8f4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6de0535d-dcf1-452d-afd4-6734c58db20e_calibrate_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6de0535d-dcf1-452d-afd4-6734c58db20e_calibrate_1039768_58.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/2aab8548-f9ee-42f5-9726-3fb1f283b5fc_characterizeImage_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/2aab8548-f9ee-42f5-9726-3fb1f283b5fc_characterizeImage_1039768_58.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa6d5e83f3ebeea6e8dd188834af52f with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 9e75749e9b3e30a692237eb525870abd parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f9128e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c57107db-ddde-4d87-a65f-d24f223ee2de_calibrate_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c57107db-ddde-4d87-a65f-d24f223ee2de_calibrate_1039770_48.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4fa7f347-445a-4709-b1b8-fc0659ee8cdf_characterizeImage_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4fa7f347-445a-4709-b1b8-fc0659ee8cdf_characterizeImage_1039770_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87c02de5b7c4ca66568e1ca9167bfc0c with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 86d030a57eedba7fe8a24789cfffbc14 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/66d3a77f-4f85-4cbe-9e16-b70b4a65e287_calibrate_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/66d3a77f-4f85-4cbe-9e16-b70b4a65e287_calibrate_1039769_58.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4dc2af93-dc20-416a-a769-c8d891b1eb4f_characterizeImage_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4dc2af93-dc20-416a-a769-c8d891b1eb4f_characterizeImage_1039769_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 161806d71284d6cdee7897630efff126 with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 6ab6b11c3c7482efd0486a7acc883a88 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/0c9e96ac-142e-48c0-9084-9a11850f2ec7_calibrate_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/0c9e96ac-142e-48c0-9084-9a11850f2ec7_calibrate_1039770_55.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1f90a765-9460-4db2-a6a8-a562c75bd406_characterizeImage_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/1f90a765-9460-4db2-a6a8-a562c75bd406_characterizeImage_1039770_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:39 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017b84588bb47f3a4d0524b7ddaeb092 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3f0fedcc-2d98-4620-a587-6283119d3e37_calibrate_1039673_19.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3f0fedcc-2d98-4620-a587-6283119d3e37_calibrate_1039673_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c617e20cdc607be9f73f88b4bb6945cd with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2f347c0a-d2e0-4b2c-9823-eeaa2729692e_calibrate_1039674_37.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2f347c0a-d2e0-4b2c-9823-eeaa2729692e_calibrate_1039674_37.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d636eb396963e2db26354049da3fc34 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 984772eb2dab35d9859576acf0a85435 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a81c64fb-e740-4bcd-bd61-dd2a2449a005_calibrate_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a81c64fb-e740-4bcd-bd61-dd2a2449a005_calibrate_1039674_33.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da06536f-1ccd-4fef-aa4e-2422646f00d6_characterizeImage_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/da06536f-1ccd-4fef-aa4e-2422646f00d6_characterizeImage_1039674_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e202f99f2016a9a5d839d92a0870951 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 10ff8f68167b35c2dccdac8b3b7047ff parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/92ba83b9-9fa4-41ae-8c41-3a523a79c490_calibrate_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/92ba83b9-9fa4-41ae-8c41-3a523a79c490_calibrate_1039674_39.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5d0e85e1-cd8c-471b-a6cf-e1d51e88cfec_characterizeImage_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/5d0e85e1-cd8c-471b-a6cf-e1d51e88cfec_characterizeImage_1039674_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:40 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 292 parsl.dataflow.strategy DEBUG: Executor multi has 292 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9cfbe29bbc2444969fb013b422391c4 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash cdcc3fc9b9c3ad98bd0ca8cbc27831ac parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1accd337-b526-45cf-83e6-7c291db89538_calibrate_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1accd337-b526-45cf-83e6-7c291db89538_calibrate_1039673_52.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cfd347f2-062e-499a-b91f-f8c7249bcb9b_characterizeImage_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/cfd347f2-062e-499a-b91f-f8c7249bcb9b_characterizeImage_1039673_52.stderr 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85b1b0af0af2364ff369fe4441951d8d with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 156c9673e879eb45b5a76cb4db494ea3 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5fce390-3fa4-4b4f-8e0c-d4c2316b8aba_calibrate_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5fce390-3fa4-4b4f-8e0c-d4c2316b8aba_calibrate_1039768_38.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5936482b-62b0-4b05-b5f6-5f13a009938b_characterizeImage_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5936482b-62b0-4b05-b5f6-5f13a009938b_characterizeImage_1039768_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abd8155bca964e2af15dfb6c418fe9fe with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e0263606-0d58-4325-98e4-e548ccc85456_calibrate_1039673_17.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e0263606-0d58-4325-98e4-e548ccc85456_calibrate_1039673_17.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2831e5d31d27547cd390e4373f808071 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d4bbe42-fba4-4549-b6c0-ec3f920e43c3_calibrate_1039672_21.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d4bbe42-fba4-4549-b6c0-ec3f920e43c3_calibrate_1039672_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:41 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d47c9d3dd246c02d422b4b9c710a7fe1 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1bfa6e37-810f-4333-8f86-c2233decab4c_calibrate_1039672_32.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1bfa6e37-810f-4333-8f86-c2233decab4c_calibrate_1039672_32.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14ef0666ed3690a08eed2431758c5e19 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4ce6af6c-f6f2-4cd9-99ea-4c060ccf3a18_calibrate_1039674_1.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4ce6af6c-f6f2-4cd9-99ea-4c060ccf3a18_calibrate_1039674_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f8f59cc9323f5bf6c625469d7b345b with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash cbb812a4b03b31e58bde236b55e1f609 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5a6d01cf-78a4-40ef-8649-5bb4d9dc8405_calibrate_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5a6d01cf-78a4-40ef-8649-5bb4d9dc8405_calibrate_1039769_38.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/daf7400f-7678-447d-b769-4400811b2541_characterizeImage_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/daf7400f-7678-447d-b769-4400811b2541_characterizeImage_1039769_38.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49f7ea9c35e984fde3b2e4dd836dcf61 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/d5d618da-43f5-4aa7-abd0-e2ec176f126f_calibrate_1039673_33.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/d5d618da-43f5-4aa7-abd0-e2ec176f126f_calibrate_1039673_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:42 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:43 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:43 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0815193b285d9eabfefeb9de67c71f08 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9a809f4e-850a-4138-8e2a-d226b2810dda_calibrate_1039674_52.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9a809f4e-850a-4138-8e2a-d226b2810dda_calibrate_1039674_52.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c0a2ebae9283419cfdcbd64bc3ea08a with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d96d720c-d861-41e4-b362-71a783e09dc7_calibrate_1039672_18.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d96d720c-d861-41e4-b362-71a783e09dc7_calibrate_1039672_18.stderr parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b2c949731fe88e09b1b8c48baaff913 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0a69c995-5e8d-4bf0-936c-7e5bac0a1364_calibrate_1039674_11.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0a69c995-5e8d-4bf0-936c-7e5bac0a1364_calibrate_1039674_11.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd0a75906902aaa7101781374d90acdd with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/55a38d70-5476-433e-bf9b-d14baf363d66_calibrate_1039672_57.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/55a38d70-5476-433e-bf9b-d14baf363d66_calibrate_1039672_57.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd58f851a044e976c9c91d1e68911406 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0156d1ca-ad77-488a-865d-17dbbebb6975_calibrate_1039672_16.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0156d1ca-ad77-488a-865d-17dbbebb6975_calibrate_1039672_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:43 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af4fe94591884edb1605b82265174114 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/79ebdfaf-a3b3-4528-8956-556b04330997_calibrate_1039674_10.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/79ebdfaf-a3b3-4528-8956-556b04330997_calibrate_1039674_10.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a2bf1ee7835842d1ad07f3fde76d4a with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/515a9f31-8138-48f1-9bf7-49155084aec7_calibrate_1039674_57.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/515a9f31-8138-48f1-9bf7-49155084aec7_calibrate_1039674_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:43 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e681d233c2232bc0cb1504df02f81a9 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 4f15ba3bd010341f95c49fa971db3807 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/411767ec-c616-4251-bc14-90a16ca7a5ce_calibrate_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/411767ec-c616-4251-bc14-90a16ca7a5ce_calibrate_1039770_43.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bc2c99fe-acc0-46e1-8954-5f186793039f_characterizeImage_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bc2c99fe-acc0-46e1-8954-5f186793039f_characterizeImage_1039770_43.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0aa003d2fdfe67e2f47355e77fb86a49 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3a8b6e4e-e937-4ab0-a89d-5b8f25d663e9_calibrate_1039673_6.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3a8b6e4e-e937-4ab0-a89d-5b8f25d663e9_calibrate_1039673_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:43 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6827d60bce38eddd6434a6c6f7fbdb1 with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash f672591f5b8a3f35bbdb619aa77766c0 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f921f64e-078c-45e1-b003-60ecf94d559b_calibrate_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f921f64e-078c-45e1-b003-60ecf94d559b_calibrate_1039672_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/5bed9fac-b870-4ebb-8f85-d39d8978cc98_characterizeImage_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/5bed9fac-b870-4ebb-8f85-d39d8978cc98_characterizeImage_1039672_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 046be6b64817c3d97c6bf6e4a6c909b6 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash befd76c4245e2d3698a1acffa5638c89 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/82db887a-b2b2-4e9a-8ba3-c3f15d7b6945_calibrate_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/82db887a-b2b2-4e9a-8ba3-c3f15d7b6945_calibrate_1039770_12.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/640e52a7-9885-4302-891a-9ebeb0cad2a4_characterizeImage_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/640e52a7-9885-4302-891a-9ebeb0cad2a4_characterizeImage_1039770_12.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44414b64146efb44358343e50c11b715 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/72052f47-0277-4a19-8da6-ccd5166dad50_calibrate_1039769_43.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/72052f47-0277-4a19-8da6-ccd5166dad50_calibrate_1039769_43.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee819631b74025ae56e0ed71adf607fa with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c33c88cd-dced-4cd7-92b7-1ef6c1652bf2_calibrate_1039673_11.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c33c88cd-dced-4cd7-92b7-1ef6c1652bf2_calibrate_1039673_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:44 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cabdc3ab43ca026e44624e543a98be0b with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash df53dd7e35acac968d31d39ab97fc5e3 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f7c3926b-e68a-4caf-8aaa-737d501ef78f_calibrate_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f7c3926b-e68a-4caf-8aaa-737d501ef78f_calibrate_1039768_54.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4591304f-8a48-4316-9102-54f2bb23de73_characterizeImage_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/4591304f-8a48-4316-9102-54f2bb23de73_characterizeImage_1039768_54.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fbcfd50cdfbe992b767722c4cae944e with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/812099fc-1c3a-4b88-84ff-388faa834e26_calibrate_1039673_55.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/812099fc-1c3a-4b88-84ff-388faa834e26_calibrate_1039673_55.stderr parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04c1182e3afbe02516d595dfbf61cd7c with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d9377a67-dbbf-4481-93d8-5da66691b0cd_calibrate_1039674_46.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d9377a67-dbbf-4481-93d8-5da66691b0cd_calibrate_1039674_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2529f37a0929510ad417c724bdf75e with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 03c133a745ca36eb1bf28ca1128b7c47 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec5b362b-2745-4153-ad3e-22939ab07631_calibrate_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec5b362b-2745-4153-ad3e-22939ab07631_calibrate_1039768_47.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/3f5ce35b-1ae5-4f05-a2b5-8a7f8a67f0c5_characterizeImage_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/3f5ce35b-1ae5-4f05-a2b5-8a7f8a67f0c5_characterizeImage_1039768_47.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1405bcff8a068a8afd1ae9b52654f769 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1e23e149-e3f8-4ddb-9fec-8663906cd9c6_calibrate_1039672_20.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1e23e149-e3f8-4ddb-9fec-8663906cd9c6_calibrate_1039672_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f6154bd3671c6404dad7fe1e1b2b48 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 0dffaed758690674eb4f7a5181f06be2 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e381940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/baedfe59-4048-484e-b44b-c2a09da9d495_calibrate_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/baedfe59-4048-484e-b44b-c2a09da9d495_calibrate_1039770_57.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/7b497ac1-15ce-4f24-b47d-169727dc43fe_characterizeImage_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/7b497ac1-15ce-4f24-b47d-169727dc43fe_characterizeImage_1039770_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d678953ad4cec5bf54efe2d8f4627aae with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/41dc1f33-bed4-4ca3-b669-92cf8c2dd697_calibrate_1039674_43.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/41dc1f33-bed4-4ca3-b669-92cf8c2dd697_calibrate_1039674_43.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce00c03ed0555ab4758d6a8769656124 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6e4fb53b-539d-4e00-a874-dc8ac39f19f2_calibrate_1039672_39.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6e4fb53b-539d-4e00-a874-dc8ac39f19f2_calibrate_1039672_39.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb74265a235cebb32fae25c2dfc71ee9 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash b5f4abb0c7dadef1086ebbd8ef88c44f parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c0e5608b-fbf6-434f-ab6f-359099d4e7b7_calibrate_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c0e5608b-fbf6-434f-ab6f-359099d4e7b7_calibrate_1039769_56.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/62899b5c-ce46-4ab9-a028-76362d400814_characterizeImage_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/62899b5c-ce46-4ab9-a028-76362d400814_characterizeImage_1039769_56.stderr parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 129a65e363f2ed04d2d0376b88c94c11 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/783e5192-11f6-4f3f-b753-0bdb7148a0ad_calibrate_1039674_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/783e5192-11f6-4f3f-b753-0bdb7148a0ad_calibrate_1039674_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73fe4a359cba7e878bf2c2b9cf61b9c7 with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/08ec79e3-5fb0-4d82-9dbd-c4bc82c3a534_calibrate_1039672_29.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/08ec79e3-5fb0-4d82-9dbd-c4bc82c3a534_calibrate_1039672_29.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 768234d9cff5c34bb1403af25c4a8265 with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5b0ff4ca-4d8b-4628-8371-82716fd81468_calibrate_1039769_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5b0ff4ca-4d8b-4628-8371-82716fd81468_calibrate_1039769_31.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4334fb25e02453becc39c17b2113b81 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5b2f8ca2-3790-4195-bbad-78cce3baf141_calibrate_1039673_46.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5b2f8ca2-3790-4195-bbad-78cce3baf141_calibrate_1039673_46.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4dc131c646d846495a895fa823cda78 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c01ec5b2-4295-483b-9cc5-6600c591f9a4_calibrate_1039673_45.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/c01ec5b2-4295-483b-9cc5-6600c591f9a4_calibrate_1039673_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:45 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4b6f4ad6408e257cd6904dada9f611b with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/df915a90-ce60-46b0-a0aa-32ecbf3473b2_calibrate_1039768_46.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/df915a90-ce60-46b0-a0aa-32ecbf3473b2_calibrate_1039768_46.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7db5c203bc3b2804a8f61726933d8c2 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/caec8f16-76fb-4fd7-9b3d-40647d5b7dd2_calibrate_1039674_51.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/caec8f16-76fb-4fd7-9b3d-40647d5b7dd2_calibrate_1039674_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1574cbb08a4adc8d505549bf46281724 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash f79338604acff9993c38193841ddb56e parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a765cd63-de86-4e50-b28e-3e2fdabf098f_calibrate_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a765cd63-de86-4e50-b28e-3e2fdabf098f_calibrate_1039672_54.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3f7db7ff-3eaf-4b60-992f-fe26fc3a10ee_characterizeImage_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3f7db7ff-3eaf-4b60-992f-fe26fc3a10ee_characterizeImage_1039672_54.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 749395820f460226df75e32a439973aa with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 153a79983aaf9262448ce4b06cfe19b9 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d2255a33-7d4e-4e35-9fdf-4b2ab1ed3230_calibrate_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d2255a33-7d4e-4e35-9fdf-4b2ab1ed3230_calibrate_1039769_16.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/216c9984-6a11-4d93-991c-abf774f8861c_characterizeImage_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/216c9984-6a11-4d93-991c-abf774f8861c_characterizeImage_1039769_16.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbe564406d23808faea0e563c7bf5fa with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/107de102-9f0c-45cd-ae78-ab4aaf0421a8_calibrate_1039673_42.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/107de102-9f0c-45cd-ae78-ab4aaf0421a8_calibrate_1039673_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d24654ee64c110620e7dd960e7ffe67a with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 392a172e2a04254168cb1e1957a25c29 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/991e745d-0f22-4d17-9f68-a8f1214ec459_calibrate_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/991e745d-0f22-4d17-9f68-a8f1214ec459_calibrate_1039769_55.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d793723-5fa3-49ee-a850-8e87e15a6484_characterizeImage_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/8d793723-5fa3-49ee-a850-8e87e15a6484_characterizeImage_1039769_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:46 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d027f0d89879f82f2167e1418f03556 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d0b55f6-b0b2-4686-ba61-7c43f966b1c2_calibrate_1039674_41.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d0b55f6-b0b2-4686-ba61-7c43f966b1c2_calibrate_1039674_41.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c4aeec3f772e364986b7d82c9fefda7 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash be5591ff71eef427c2853a3b20a1ee3b parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f204060d-58aa-4d99-a3d4-5e1e0b9fbdc5_calibrate_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f204060d-58aa-4d99-a3d4-5e1e0b9fbdc5_calibrate_1039674_26.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b1c428a8-98fa-4120-8636-1029d63dacff_characterizeImage_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b1c428a8-98fa-4120-8636-1029d63dacff_characterizeImage_1039674_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b93d161c1b72cb23a225b35b0735a2b with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ae81b794-ef70-4ccd-b66a-78251ad388f3_calibrate_1039674_17.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/ae81b794-ef70-4ccd-b66a-78251ad388f3_calibrate_1039674_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5763f669bba6b9f06de02d336bf18d01 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 6759e5fdcfbf8a9d73ed8f9c6967e076 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/cdcea881-1f38-48d1-ba46-34be9ca08c6e_calibrate_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/cdcea881-1f38-48d1-ba46-34be9ca08c6e_calibrate_1039768_7.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ead6dfb7-27b5-424a-9cdc-709c28c49dd9_characterizeImage_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ead6dfb7-27b5-424a-9cdc-709c28c49dd9_characterizeImage_1039768_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:47 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41553aebe8cfa3d66e805ee7b79f0b3 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0838d5c9-bdc5-431e-a1f8-96b1621ff74d_calibrate_1039672_53.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0838d5c9-bdc5-431e-a1f8-96b1621ff74d_calibrate_1039672_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a2c6ac1106f22f020c79bcb06948ead with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 8e3f844a389ea545947ad4b5b0d7a71b parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/77fde3fe-e3bd-4835-a6d0-d413195c7105_calibrate_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/77fde3fe-e3bd-4835-a6d0-d413195c7105_calibrate_1039768_23.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a0879060-dcd3-4095-b7f5-bf0f5a63fb7c_characterizeImage_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/a0879060-dcd3-4095-b7f5-bf0f5a63fb7c_characterizeImage_1039768_23.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1ff7e234c225c63b86be1707a70695e with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 316749764e5a8b0aa5bd2d8db95c10fd parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/85531c43-e07a-4f5f-a720-cca0172ab61a_calibrate_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/85531c43-e07a-4f5f-a720-cca0172ab61a_calibrate_1039673_5.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/adf4f691-5656-4077-a9bc-dd807989def9_characterizeImage_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/adf4f691-5656-4077-a9bc-dd807989def9_characterizeImage_1039673_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0e858b81d337fe2a9a4e6f3a83a34f1 with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7c3c853c-f9f8-4263-a2df-efc07b08509f_calibrate_1039672_33.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7c3c853c-f9f8-4263-a2df-efc07b08509f_calibrate_1039672_33.stderr parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33099a19c91e16bb992b436db3ac55e3 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/59559173-e247-47b9-bb3c-c84d6d7e55b6_calibrate_1039672_1.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/59559173-e247-47b9-bb3c-c84d6d7e55b6_calibrate_1039672_1.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4b9340ee4012e25717d43a9e7cc0eb9 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2c1d0059-23ff-4f10-bc3f-430ef94e9182_calibrate_1039770_30.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2c1d0059-23ff-4f10-bc3f-430ef94e9182_calibrate_1039770_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25d939a11722cf977bb6a5c65febf69 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 4f25ac29de606652a9d4fb0a737c45bb parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8740d17a-6fa3-42ad-9bc7-ed687f409a49_calibrate_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8740d17a-6fa3-42ad-9bc7-ed687f409a49_calibrate_1039672_5.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e172a2e-1003-4d4f-8de4-d0b4bcdf3a96_characterizeImage_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/7e172a2e-1003-4d4f-8de4-d0b4bcdf3a96_characterizeImage_1039672_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 710d38a0118a10acf954acb0a1c0cc23 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/72a74625-c8ec-4587-a327-c2e0d89f6d2d_calibrate_1039770_52.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/72a74625-c8ec-4587-a327-c2e0d89f6d2d_calibrate_1039770_52.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 344a97c984eb590c66ebd1bb79b49c2a with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash 1b02b1a891401efe03b77d80cfa8cf8a parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9b843319-4a82-4b37-a419-e82ecae9b49b_calibrate_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9b843319-4a82-4b37-a419-e82ecae9b49b_calibrate_1039768_5.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b9d7197f-8e12-4286-b3c1-c08af633edaa_characterizeImage_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/b9d7197f-8e12-4286-b3c1-c08af633edaa_characterizeImage_1039768_5.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7972a238dcef9d434291d33fa1306490 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0ea1867f-d42a-4c44-9c4c-e958a1ef0972_calibrate_1039674_35.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/0ea1867f-d42a-4c44-9c4c-e958a1ef0972_calibrate_1039674_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9300a515b3c2918645c46caf0e27b516 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash b46f304273ef49ffb0cf2c6640b61724 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5d290c0b-81b1-479e-88e3-9c0d6ce9de18_calibrate_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5d290c0b-81b1-479e-88e3-9c0d6ce9de18_calibrate_1039768_20.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a66e2a5-c4f7-4920-9efe-b0264f3e8735_characterizeImage_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0a66e2a5-c4f7-4920-9efe-b0264f3e8735_characterizeImage_1039768_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:48 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0c0856f1b1d68652471a0d176d07d37 with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash e41067d9b489ac460c37bf4c6bab39d7 parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/48fbef5c-67b5-4d0d-96f3-0ab4108208a6_calibrate_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/48fbef5c-67b5-4d0d-96f3-0ab4108208a6_calibrate_1039673_59.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a5aeb4e1-1a4c-4f43-871f-05d2b3809c51_characterizeImage_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/a5aeb4e1-1a4c-4f43-871f-05d2b3809c51_characterizeImage_1039673_59.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43564120c112f876b35cc1814c9ecfce with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ffef57ce-03c9-4829-b0d5-ab75866cce5f_calibrate_1039672_3.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ffef57ce-03c9-4829-b0d5-ab75866cce5f_calibrate_1039672_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e07767107ebb0953defaf06e1a920bb with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ac6e8bc5-9597-471c-8202-563d597368bf_calibrate_1039672_25.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/ac6e8bc5-9597-471c-8202-563d597368bf_calibrate_1039672_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc8ad9df023ef62d1f2bb47857954ad7 with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 52b63679f487bde4ade59afdf3038698 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4b12ce15-ee5d-4a68-bc10-7a688f3d460f_calibrate_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4b12ce15-ee5d-4a68-bc10-7a688f3d460f_calibrate_1039768_27.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d997cd36-1385-47a5-8047-3579e3cbbe78_characterizeImage_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/d997cd36-1385-47a5-8047-3579e3cbbe78_characterizeImage_1039768_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dcb2cf0e93f13c18045b527cf95fd06 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 3d2b73227af23b5270ea33f100d5e03b parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3175324b-5e53-436c-bff8-5475e894d297_calibrate_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3175324b-5e53-436c-bff8-5475e894d297_calibrate_1039673_12.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c664ebdb-8626-4050-91a2-b6b0de715653_characterizeImage_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/c664ebdb-8626-4050-91a2-b6b0de715653_characterizeImage_1039673_12.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e86ed5f99fe67410c1a4ca93dca11cbb with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a6394304-6a53-44e8-92a9-294555ebb1b5_calibrate_1039674_54.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a6394304-6a53-44e8-92a9-294555ebb1b5_calibrate_1039674_54.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e487b1bd61874cae5aff908696e70e85 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/62519e82-dcb8-440a-b073-990ebc5ac4aa_calibrate_1039673_57.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/62519e82-dcb8-440a-b073-990ebc5ac4aa_calibrate_1039673_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:49 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f2010490dcb6491b3614220aabc8684 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1dce5dfe-18ac-43f1-8a62-da0f42cd0d72_calibrate_1039672_59.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/1dce5dfe-18ac-43f1-8a62-da0f42cd0d72_calibrate_1039672_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49ccb8747a64984e12388f5420b66c4c with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 4752ddda3c70b9e28e953fbf8b5cc45d parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3827a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3484d90f-346e-41e4-940d-4f4cdcaeca09_calibrate_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3484d90f-346e-41e4-940d-4f4cdcaeca09_calibrate_1039770_47.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2e17e90a-f61f-4126-9d86-fce3f3a9395c_characterizeImage_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/2e17e90a-f61f-4126-9d86-fce3f3a9395c_characterizeImage_1039770_47.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9136f42c35227d03289858ca276484d with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a894960d-0356-4d3f-94f9-8b10c12ae5ce_calibrate_1039672_28.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a894960d-0356-4d3f-94f9-8b10c12ae5ce_calibrate_1039672_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fe6d598f7d34c66405c47f067fd36e7 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 4530f9b2e72293410a7fe708b61f118a parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6974ff9a-1a0f-4fb9-9fd5-f186b5274e7c_calibrate_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6974ff9a-1a0f-4fb9-9fd5-f186b5274e7c_calibrate_1039674_25.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/decbcd64-e7c6-4152-aa79-9e6967caa648_characterizeImage_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/decbcd64-e7c6-4152-aa79-9e6967caa648_characterizeImage_1039674_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7f0e5328f8ffb235c435be54b8a2a7 with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 8c17125001cc6eed268c8d21b0f94d58 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bd3d5656-adfb-4820-af8e-6f578443fa5f_calibrate_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bd3d5656-adfb-4820-af8e-6f578443fa5f_calibrate_1039768_26.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/69f517e6-c5b9-4d8b-b1f5-6a4031d3dcf9_characterizeImage_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/69f517e6-c5b9-4d8b-b1f5-6a4031d3dcf9_characterizeImage_1039768_26.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd0464dac16bdb987dd39ad439885a9 with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/e0c3929b-23b3-46ea-a97c-378c8bc500db_calibrate_1039674_55.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/e0c3929b-23b3-46ea-a97c-378c8bc500db_calibrate_1039674_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:50 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68692e1ec07fee151d000551b0a55050 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash bf01f916ee9e8ae7d23aa2c870687f46 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c6c40178-9d72-4a85-8a2b-055069f81c9c_calibrate_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c6c40178-9d72-4a85-8a2b-055069f81c9c_calibrate_1039770_50.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54ab28f6-949e-4780-9796-36da3e11d1dc_characterizeImage_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/54ab28f6-949e-4780-9796-36da3e11d1dc_characterizeImage_1039770_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18aeb2da3ec0a6bd402ae11191eda86f with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8ca3c456-7bc9-4a79-a465-4a43a99421ef_calibrate_1039674_13.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/8ca3c456-7bc9-4a79-a465-4a43a99421ef_calibrate_1039674_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:51 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfddfe2e9c0d1e990cc504a290c4978e with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/2655ee26-2876-4cbd-8fa6-0672a68f4b8a_calibrate_1039673_8.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/2655ee26-2876-4cbd-8fa6-0672a68f4b8a_calibrate_1039673_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da769b60107f8fcecbe09df638df2162 with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/77fbcba7-f7e7-4da8-af9c-12b110782355_calibrate_1039673_28.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/77fbcba7-f7e7-4da8-af9c-12b110782355_calibrate_1039673_28.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b33c8e30b105fb0004782cd5ef41f604 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash ec99a969d389ebdf821221036f1b606a parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3811c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/26a2ca98-0899-42df-9d1b-3a69d8092a7a_calibrate_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/26a2ca98-0899-42df-9d1b-3a69d8092a7a_calibrate_1039674_62.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/ad9a6a88-2436-448d-bc2a-e4c4b4586a13_characterizeImage_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/ad9a6a88-2436-448d-bc2a-e4c4b4586a13_characterizeImage_1039674_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa31c1a370c9b277e4035560c83e4b9a with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash cd3e35bf8e504c9863186420eb6702bd parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7dad39df-89ef-48c7-86ee-66a83cd03880_calibrate_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7dad39df-89ef-48c7-86ee-66a83cd03880_calibrate_1039673_41.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/135a5858-6684-46bc-87b5-ba3bdd7dd7a4_characterizeImage_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/135a5858-6684-46bc-87b5-ba3bdd7dd7a4_characterizeImage_1039673_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:52 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c690d415efc70f1906353905eaf699f8 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash b0970382419636ee77f01cb617cbeeae parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea6d6f27-f247-4aad-a87c-6d25f39bfe05_calibrate_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea6d6f27-f247-4aad-a87c-6d25f39bfe05_calibrate_1039769_40.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f53ea1d1-371e-4944-862e-f3a33ca6c44b_characterizeImage_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f53ea1d1-371e-4944-862e-f3a33ca6c44b_characterizeImage_1039769_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cf3dc8eb0d663ff3ca36b1239b0edcc with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash e9dd9dd9d884602602817bc7aa4f3aa1 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/1959221c-d848-47df-b5ee-da3581b8fc91_calibrate_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/1959221c-d848-47df-b5ee-da3581b8fc91_calibrate_1039768_45.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/63f2c2c7-f887-4ca2-8bd4-21ca10dec3af_characterizeImage_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/63f2c2c7-f887-4ca2-8bd4-21ca10dec3af_characterizeImage_1039768_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:53 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:54 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8855458028dd0d03bbdfa059cf6548ac with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2e019ceb-6bdb-4402-947e-5beaa39e787b_calibrate_1039770_11.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2e019ceb-6bdb-4402-947e-5beaa39e787b_calibrate_1039770_11.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cbdbf6aeb6f140524356ccedabe6da6 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/04dde255-6a4e-42b1-9c8b-34ca47eb37cd_calibrate_1039672_58.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/04dde255-6a4e-42b1-9c8b-34ca47eb37cd_calibrate_1039672_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:54 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:54 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afa0dd30585772157266e3e34714ab19 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b8edb12-eef0-4d74-a78e-5305c98ded4e_calibrate_1039673_60.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b8edb12-eef0-4d74-a78e-5305c98ded4e_calibrate_1039673_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:54 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe24c89116654ffcd85e7356458f030 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash adc269a97b0a306dc889aec65f260203 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/70df13f1-da17-4f3f-bb18-fd85072221a2_calibrate_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/70df13f1-da17-4f3f-bb18-fd85072221a2_calibrate_1039769_32.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c0b4bbb8-e132-4a27-ab74-070c6ec0fad4_characterizeImage_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/c0b4bbb8-e132-4a27-ab74-070c6ec0fad4_characterizeImage_1039769_32.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854355745ca48ff5f0dbe48a66a2122c with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/78f988d4-a76f-4e5d-b663-60bde10e73ab_calibrate_1039673_20.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/78f988d4-a76f-4e5d-b663-60bde10e73ab_calibrate_1039673_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:54 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5608918ef5624392e5e30aedbeeeff9 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d37f3a1-9685-41b6-beaf-ff6c8d9d1aa1_calibrate_1039674_27.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/7d37f3a1-9685-41b6-beaf-ff6c8d9d1aa1_calibrate_1039674_27.stderr parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 874d28c58184a3c2f1d16d52462785fa with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dc0a3f30-3df0-4523-b19d-4ee973beb246_calibrate_1039673_53.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/dc0a3f30-3df0-4523-b19d-4ee973beb246_calibrate_1039673_53.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20c4749eaafda22c9658df4cc6d695d with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/20272cbb-80a3-414d-91af-efcffe25a1ae_calibrate_1039672_31.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/20272cbb-80a3-414d-91af-efcffe25a1ae_calibrate_1039672_31.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c19f25ad86b9f59bb86013c446838c0 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7073cbbe-2b43-4798-9033-a68082406c6f_calibrate_1039769_48.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7073cbbe-2b43-4798-9033-a68082406c6f_calibrate_1039769_48.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0789a807e5f644a3e15b125fdeb6452a with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash cac254fbb8dc5c6784b96e25e03b20c5 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ffc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/122e7af4-d3c0-486f-a487-63e814ee6c8d_calibrate_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/122e7af4-d3c0-486f-a487-63e814ee6c8d_calibrate_1039768_11.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdae1d56-5462-437d-81ce-47b3b2d63ba1_characterizeImage_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdae1d56-5462-437d-81ce-47b3b2d63ba1_characterizeImage_1039768_11.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bdd73091609b0893bd061b35132808d with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 60202f1e1f3869774b31f5384846b0e6 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6a1371b3-80c2-4ca9-960a-c54ef4f4cec7_calibrate_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6a1371b3-80c2-4ca9-960a-c54ef4f4cec7_calibrate_1039770_6.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bfc7a977-e5e2-4f20-8b56-26997df86415_characterizeImage_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/bfc7a977-e5e2-4f20-8b56-26997df86415_characterizeImage_1039770_6.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a3926123ee5b04806b755fdc395d7f with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 3657ee8e888c1cd93cdb1ee52faeefd5 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3805e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/19a0d367-c72b-4feb-99f7-2cc1b7242548_calibrate_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/19a0d367-c72b-4feb-99f7-2cc1b7242548_calibrate_1039768_40.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/540be8eb-3066-4ae7-b3e6-9cc910939432_characterizeImage_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/540be8eb-3066-4ae7-b3e6-9cc910939432_characterizeImage_1039768_40.stderr parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78934d0633227cea138003e52261001c with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/064ee654-a579-4698-90f2-a77fe924a150_calibrate_1039768_25.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/064ee654-a579-4698-90f2-a77fe924a150_calibrate_1039768_25.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61c76af837b70c64c1eb0d3104fc5f5d with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a8d70291-f947-48fb-8dbc-444e5dc8e443_calibrate_1039768_13.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a8d70291-f947-48fb-8dbc-444e5dc8e443_calibrate_1039768_13.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe693c55571fca439a549e390cd8113b with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash e59c18c3d705e376aef7917418545257 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3837e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/061f6a16-46df-4292-81f7-1959428df6dd_calibrate_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/061f6a16-46df-4292-81f7-1959428df6dd_calibrate_1039768_28.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0349da11-5784-4dae-b050-2675f0dfe144_characterizeImage_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/0349da11-5784-4dae-b050-2675f0dfe144_characterizeImage_1039768_28.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5d95d86b6885eb265ecbdf3d3320f3 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash c6337704c58d5cad6c5c8d502e65d138 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8585b4c3-d6cf-45c8-9dfc-99539e2d7e6f_calibrate_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8585b4c3-d6cf-45c8-9dfc-99539e2d7e6f_calibrate_1039770_28.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/49cb8a90-c9cf-4a2e-b223-0aa6a72221bb_characterizeImage_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/49cb8a90-c9cf-4a2e-b223-0aa6a72221bb_characterizeImage_1039770_28.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6396de9bdd251d86b3f77a71f7714d50 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fb3ee3e0-db25-4414-9686-4d0faed72d81_calibrate_1039770_35.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fb3ee3e0-db25-4414-9686-4d0faed72d81_calibrate_1039770_35.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f0a972d6caa90b5e68204bc38a2ff8 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash 155206f5b72c69a9f75c38fac49c81a3 parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/df5fd3a9-9bbc-4094-850e-fd56ee24416e_calibrate_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/df5fd3a9-9bbc-4094-850e-fd56ee24416e_calibrate_1039674_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b503ee85-f6b9-402e-96a5-f0ab5d025df5_characterizeImage_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b503ee85-f6b9-402e-96a5-f0ab5d025df5_characterizeImage_1039674_29.stderr parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cc41cb4d2ce936fa47bc7c459416e62 with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ddae20fa-bc2e-4fb0-892e-c03317757554_calibrate_1039673_23.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ddae20fa-bc2e-4fb0-892e-c03317757554_calibrate_1039673_23.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4579df4135a335daa5197425c29d5a5 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/0521abb8-a06b-40b3-a5c6-5d3ed75c995a_calibrate_1039768_15.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/0521abb8-a06b-40b3-a5c6-5d3ed75c995a_calibrate_1039768_15.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1653f3a4609b9ca35e2ee072f917546 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c3747be-efb1-4475-a002-a478fa70339e_calibrate_1039673_38.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c3747be-efb1-4475-a002-a478fa70339e_calibrate_1039673_38.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 057a92a02ed88565aa7f73a621e26e9b with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 2e4f6e320dba8859fe5013a1a51e0622 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/862ed1af-e35f-462f-8131-3d391734d74b_calibrate_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/862ed1af-e35f-462f-8131-3d391734d74b_calibrate_1039672_14.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ceed348-2929-42a2-817e-2b6dab21df23_characterizeImage_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/3ceed348-2929-42a2-817e-2b6dab21df23_characterizeImage_1039672_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:56 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a76ca5159e313af2abced1db52a3ed64 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash 05057aeffe36deb0174dac1574dbab5f parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ffec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4958bdfd-4cc4-4535-adae-eb44d603a6c4_calibrate_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4958bdfd-4cc4-4535-adae-eb44d603a6c4_calibrate_1039674_15.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/bcdee720-2ff7-4945-ab70-7b4a811e2a43_characterizeImage_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/bcdee720-2ff7-4945-ab70-7b4a811e2a43_characterizeImage_1039674_15.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b843d5aa1d3483b8f949647550c76e63 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 4417f56fddfce2f19117f05ffb1fe3ab parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5368c92-8c3f-4e33-b009-caaeefb93a44_calibrate_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5368c92-8c3f-4e33-b009-caaeefb93a44_calibrate_1039768_56.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/9795f375-835b-43de-85ba-5019c85c6bdb_characterizeImage_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/9795f375-835b-43de-85ba-5019c85c6bdb_characterizeImage_1039768_56.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a2cf49aebfad90bc95094d7132dc317 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8e756723-ab97-449f-87c7-18c5ae35dbf4_calibrate_1039672_38.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8e756723-ab97-449f-87c7-18c5ae35dbf4_calibrate_1039672_38.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 083d5c3378fe613873e47dfddb732444 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/18f7b382-1d16-4d23-9e0c-55c63a760210_calibrate_1039673_30.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/18f7b382-1d16-4d23-9e0c-55c63a760210_calibrate_1039673_30.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 246ac5c581fb5d36c8e4265dafc39718 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5c0f129d-8277-4af3-87fa-fedd0f7da289_calibrate_1039768_53.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5c0f129d-8277-4af3-87fa-fedd0f7da289_calibrate_1039768_53.stderr parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13272baccd105bca7fe649a7fe934cea with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/93682c3d-bea1-478c-982a-67f29812847c_calibrate_1039674_53.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/93682c3d-bea1-478c-982a-67f29812847c_calibrate_1039674_53.stderr parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dba8adfe7d4675a9af1d5b733ec659bf with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9679cd5c-2bdd-49e2-b8e4-b442b3504876_calibrate_1039674_7.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/9679cd5c-2bdd-49e2-b8e4-b442b3504876_calibrate_1039674_7.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a0ab0be7c58141b393a9327f7aea558 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 3a61d2707d66958ab3aa940ef68acac1 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/07431680-1b20-4355-85f8-a90c637c1922_calibrate_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/07431680-1b20-4355-85f8-a90c637c1922_calibrate_1039769_53.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4c4d87c7-79ac-4d58-ba74-f2c0be05ea00_characterizeImage_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/4c4d87c7-79ac-4d58-ba74-f2c0be05ea00_characterizeImage_1039769_53.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b740939bf83e6852401e19233a300cc with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash f4e44b59f9215ffd7d81b62ee6941e0d parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea7a78a7-c893-4f94-88c6-e6bed2f21ea9_calibrate_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea7a78a7-c893-4f94-88c6-e6bed2f21ea9_calibrate_1039769_29.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3c20682e-ac46-414e-90b3-bb76ff6d5861_characterizeImage_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/3c20682e-ac46-414e-90b3-bb76ff6d5861_characterizeImage_1039769_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:41:57 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e143e8f870db7d29715bbcab36feaa with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f12ce55d-9e1e-4738-a65e-8296a55c6c04_calibrate_1039770_4.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f12ce55d-9e1e-4738-a65e-8296a55c6c04_calibrate_1039770_4.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a1a1bf87150396b03313a90ef5d438 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b2c355cc-df3d-45d7-95e9-bbefc5954472_calibrate_1039673_4.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b2c355cc-df3d-45d7-95e9-bbefc5954472_calibrate_1039673_4.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a7914c091db6e8a09afff4802ceffa2 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 4674afa9fd445b5a6670971e9deccd2e parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/79c1a4c7-f7eb-4931-b3f3-c0f57da9e902_calibrate_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/79c1a4c7-f7eb-4931-b3f3-c0f57da9e902_calibrate_1039768_9.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/91752812-65d8-49d9-b57f-86bbd89bdc97_characterizeImage_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/91752812-65d8-49d9-b57f-86bbd89bdc97_characterizeImage_1039768_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 225 parsl.dataflow.strategy DEBUG: Executor multi has 225 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 225 parsl.dataflow.strategy DEBUG: Executor multi has 225 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 374a8b1363c3e5be0f67a96eaf4d515e with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/abf55b8f-0e19-4bfb-9902-885399f24f4b_calibrate_1039768_59.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/abf55b8f-0e19-4bfb-9902-885399f24f4b_calibrate_1039768_59.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffc5be0706adec58282a39376ec36748 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 93ecf651efc1ab1ff0ab6604afd75d18 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/46668289-8208-440b-84b2-9448fb18164f_calibrate_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/46668289-8208-440b-84b2-9448fb18164f_calibrate_1039770_56.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b716fef5-5fa9-4fdd-ad80-f3d82ae7dd42_characterizeImage_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b716fef5-5fa9-4fdd-ad80-f3d82ae7dd42_characterizeImage_1039770_56.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9799356fafc093ec31dee0c7174c42ea with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5d136ede-915d-4998-a5ee-a229b6e9741a_calibrate_1039673_31.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/5d136ede-915d-4998-a5ee-a229b6e9741a_calibrate_1039673_31.stderr parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21ed6a644e9a28aa9d6df886ccbed6c9 with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ad152652-bc84-496c-ad09-facbe09383fa_calibrate_1039768_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ad152652-bc84-496c-ad09-facbe09383fa_calibrate_1039768_37.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f91b46fb72128236acb3983879f622f with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/80b11c93-7eff-46aa-93de-92c2e8543c0f_calibrate_1039672_22.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/80b11c93-7eff-46aa-93de-92c2e8543c0f_calibrate_1039672_22.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b350c2607191667173b32bfd1d126c26 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f2f7ac68-ceb6-4227-8ef3-73800ddd8b72_calibrate_1039768_60.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f2f7ac68-ceb6-4227-8ef3-73800ddd8b72_calibrate_1039768_60.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry badcc86db2cf4e23926c2ff97d8266ad with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fac0cfd4-fa40-4a40-9693-54045fdab61e_calibrate_1039672_8.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/fac0cfd4-fa40-4a40-9693-54045fdab61e_calibrate_1039672_8.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51874d72e90a7210bf95e8ad36b68264 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/2848f74b-69c4-440f-9f83-f5ef131f7697_calibrate_1039768_8.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/2848f74b-69c4-440f-9f83-f5ef131f7697_calibrate_1039768_8.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecd368078db68ac49f5dfbbc6443a185 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ca75a4d8-d78b-4298-b756-028e4d383e5d_calibrate_1039768_50.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ca75a4d8-d78b-4298-b756-028e4d383e5d_calibrate_1039768_50.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be31f21df955a899ca2edf9884ee040 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/c1858d6b-dce2-4914-b682-3e70087fe686_calibrate_1039672_19.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/c1858d6b-dce2-4914-b682-3e70087fe686_calibrate_1039672_19.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427c781ce231ab5c6fa0067582cdd0f6 with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash b79980f828e3e0b33807b2bb9621f5e6 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec6c1845-bd6a-4875-94e4-98ec8966b38a_calibrate_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec6c1845-bd6a-4875-94e4-98ec8966b38a_calibrate_1039768_52.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/48ebba0d-27d3-4734-8480-f8d3733b90f1_characterizeImage_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/48ebba0d-27d3-4734-8480-f8d3733b90f1_characterizeImage_1039768_52.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 693ca2412fc0984f7507df6cf0a0ea15 with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 498a6f1188a3578745135c1bb9a913d5 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e1eb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/df17abff-7724-4b4f-b064-7dcc39fd25d6_calibrate_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/df17abff-7724-4b4f-b064-7dcc39fd25d6_calibrate_1039769_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bb9efa5-5f78-4cd9-b6af-e4117611e83f_characterizeImage_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bb9efa5-5f78-4cd9-b6af-e4117611e83f_characterizeImage_1039769_23.stderr parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5123693aaf77d98a77035f429ce91e5a with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/b5b7a097-aab5-47e2-b9eb-3c11156d19ce_calibrate_1039672_45.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/b5b7a097-aab5-47e2-b9eb-3c11156d19ce_calibrate_1039672_45.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94c5757c6590cc83325dd2fc4096e4a5 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash f4ea5c89921f1ea2f8e4cc030d890bed parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/9febbf66-1f7b-4760-a359-fba5614aff76_calibrate_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/9febbf66-1f7b-4760-a359-fba5614aff76_calibrate_1039673_58.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/043d8722-05d9-4ce0-a6af-db81236cf69b_characterizeImage_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/043d8722-05d9-4ce0-a6af-db81236cf69b_characterizeImage_1039673_58.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeba696421695b7880430989aca8db7f with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3fa36671-61a8-4848-a4af-1918ecb1aea2_calibrate_1039768_14.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3fa36671-61a8-4848-a4af-1918ecb1aea2_calibrate_1039768_14.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43d495fb41b926d6ac3a0d2f8a9d5851 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 178ac80a2a531961b4d358cde849393e parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/64863aee-88f0-44ae-972e-1b617a37d931_calibrate_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/64863aee-88f0-44ae-972e-1b617a37d931_calibrate_1039770_15.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cb8c69fa-ebef-4cc5-9af0-2b762e320026_characterizeImage_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/cb8c69fa-ebef-4cc5-9af0-2b762e320026_characterizeImage_1039770_15.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 214 parsl.dataflow.strategy DEBUG: Executor multi has 214 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03eec8517a7072a5775bb97cfbdf608a with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/38ce0945-0a61-4639-8798-4769578928fa_calibrate_1039672_43.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/38ce0945-0a61-4639-8798-4769578928fa_calibrate_1039672_43.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60a069941ee20cebc6360f2f2fab2f0a with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fa85f0b6-aaa0-413c-87cf-78117b0396fb_calibrate_1039674_59.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fa85f0b6-aaa0-413c-87cf-78117b0396fb_calibrate_1039674_59.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c104974e4031dcbbc83d99c8e1916f0 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/699d5c66-a6e2-4a69-b5f8-0cc4c1d7120d_calibrate_1039768_18.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/699d5c66-a6e2-4a69-b5f8-0cc4c1d7120d_calibrate_1039768_18.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e1639eb3e0f45d441644539d5d713e6 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/56e56363-0726-4393-b59b-d9333ba68c46_calibrate_1039673_43.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/56e56363-0726-4393-b59b-d9333ba68c46_calibrate_1039673_43.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c813a89ffa637dc65bc6e96ba734c1f with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3f7269bf-5246-46d6-b779-2f059a3bb320_calibrate_1039769_37.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3f7269bf-5246-46d6-b779-2f059a3bb320_calibrate_1039769_37.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19e06ef2c59148a07fcab3375b440048 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 5db0af27e58d2546c7895e52ec198d48 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/eb45920c-ca53-4457-adb4-275e62ac51c2_calibrate_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/eb45920c-ca53-4457-adb4-275e62ac51c2_calibrate_1039768_34.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5bec6aea-cb92-49c4-b9dc-cfdc6c375e8a_characterizeImage_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/5bec6aea-cb92-49c4-b9dc-cfdc6c375e8a_characterizeImage_1039768_34.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2935c9e679f24daeb71ca60698ef1fe9 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 3b8e9a1e7305410724e0bfbcf091f5d3 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/50f31899-c526-44a0-9b33-63157a3988fa_calibrate_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/50f31899-c526-44a0-9b33-63157a3988fa_calibrate_1039768_30.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/7b5fb8e9-a9fb-4bbb-b86b-9521dac776f6_characterizeImage_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/7b5fb8e9-a9fb-4bbb-b86b-9521dac776f6_characterizeImage_1039768_30.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bd366601fea6fb07609c5a855bf7e7b with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 8f0268e30e19957ec01c694efae8f92b parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fab5e3bc-4efb-4f8c-bc16-d6e4cf3dc483_calibrate_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fab5e3bc-4efb-4f8c-bc16-d6e4cf3dc483_calibrate_1039674_58.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b6e3c96d-1407-423e-aacc-d2ed0bada66d_characterizeImage_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/b6e3c96d-1407-423e-aacc-d2ed0bada66d_characterizeImage_1039674_58.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae019ab654cdfb247617185ed9940579 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/30c07768-e7f2-45bf-9df6-2ad679ec70c1_calibrate_1039674_16.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/30c07768-e7f2-45bf-9df6-2ad679ec70c1_calibrate_1039674_16.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fcdfe57137ffe176d9dbf16bc64070e with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4e47a799-a116-43c9-a1ab-46bc0020f2ac_calibrate_1039674_47.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4e47a799-a116-43c9-a1ab-46bc0020f2ac_calibrate_1039674_47.stderr parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c3bf731ce20a8209331887f20c85e6 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/467834c5-f5cf-46b6-950a-8f97644d5ce6_calibrate_1039674_18.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/467834c5-f5cf-46b6-950a-8f97644d5ce6_calibrate_1039674_18.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a79ce075325ac2293c2e4ba8ae90008 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1e1a65ad-2d72-4a82-9b6a-fabac9b925ad_calibrate_1039673_44.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1e1a65ad-2d72-4a82-9b6a-fabac9b925ad_calibrate_1039673_44.stderr parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89648522eb29978e5e10cba75bbd51d3 with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash e8487576615c274008bd36074b0007bd parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e18e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d8d4adab-4b41-4890-8225-a29ae65bc984_calibrate_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d8d4adab-4b41-4890-8225-a29ae65bc984_calibrate_1039768_19.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdad1e69-59fe-4010-972d-656398c7809c_characterizeImage_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cdad1e69-59fe-4010-972d-656398c7809c_characterizeImage_1039768_19.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b24a2130c783cd595ef1b24c1b4223e with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d4943173-d773-410a-844c-d83cccced265_calibrate_1039768_44.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d4943173-d773-410a-844c-d83cccced265_calibrate_1039768_44.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5593d2a9704ba6d43a60052d292a8286 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/fa3eb287-b3e0-43f6-871b-7419375f1d5b_calibrate_1039673_10.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/fa3eb287-b3e0-43f6-871b-7419375f1d5b_calibrate_1039673_10.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e5970dd00c66db0799a71de84bc1fea with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7b928ae6-f61c-4c0c-912c-1a6311db7d74_calibrate_1039769_60.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7b928ae6-f61c-4c0c-912c-1a6311db7d74_calibrate_1039769_60.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40bf88d37c6747bbffc871ec29b768b3 with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ca8cea66-4916-4b80-9f71-ea72299b7634_calibrate_1039673_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/ca8cea66-4916-4b80-9f71-ea72299b7634_calibrate_1039673_1.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9108c1edbb52bcf5bb14353705a61c4a with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/65015bc1-5123-4070-957f-599458fb5f43_calibrate_1039672_10.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/65015bc1-5123-4070-957f-599458fb5f43_calibrate_1039672_10.stderr parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f57e67e5cdd8f52680847ac2fe9ab0 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2d1322f7-05e7-4732-ab31-3477803c0f24_calibrate_1039674_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/2d1322f7-05e7-4732-ab31-3477803c0f24_calibrate_1039674_32.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df5fec5f9132827ea558d21d7cb6385b with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/26a7df34-713a-40df-9fe2-986bf7fa19da_calibrate_1039672_6.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/26a7df34-713a-40df-9fe2-986bf7fa19da_calibrate_1039672_6.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01c99cc0894e96564666b7ecdd693678 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/11cb396c-499d-4535-835a-794849576459_calibrate_1039674_20.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/11cb396c-499d-4535-835a-794849576459_calibrate_1039674_20.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 197 parsl.dataflow.strategy DEBUG: Executor multi has 197 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7328388aa90c10e0844ba035777a641c with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash cbf24b81c4cc54023367a0cee72622ee parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/88e96da5-b09f-42e9-95ee-84c9d2d7a859_calibrate_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/88e96da5-b09f-42e9-95ee-84c9d2d7a859_calibrate_1039769_30.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e658698d-1157-4f0b-b16e-6b1bd6953e18_characterizeImage_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/e658698d-1157-4f0b-b16e-6b1bd6953e18_characterizeImage_1039769_30.stderr parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 501e6e88b942747bf06a665b6a96ad73 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5845820b-2225-4bc1-9d7d-f3c73a2aa41b_calibrate_1039770_13.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5845820b-2225-4bc1-9d7d-f3c73a2aa41b_calibrate_1039770_13.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a909a29edb7df99e206b514ba55444a with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash cbe5c3e69a209184eba20efbcbd6c014 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5e82b926-81a9-447f-b3ba-c5680b0ee621_calibrate_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5e82b926-81a9-447f-b3ba-c5680b0ee621_calibrate_1039769_15.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bf0ae2b-4b5c-46f5-b026-2d6c3f1145d6_characterizeImage_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/5bf0ae2b-4b5c-46f5-b026-2d6c3f1145d6_characterizeImage_1039769_15.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5901348ef2a74e9d08aba6aafba3b1e with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 1cb3ed18d47814f863b7625738674dba parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bf1ce586-c861-41b9-9913-23986504d43e_calibrate_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bf1ce586-c861-41b9-9913-23986504d43e_calibrate_1039769_18.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/378768ce-924b-43a3-b6a4-6b833e559ebb_characterizeImage_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/378768ce-924b-43a3-b6a4-6b833e559ebb_characterizeImage_1039769_18.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d5bc6cae074d33ef898cc37407e392e with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5efd77f6-c915-41d9-8832-395783e38af2_calibrate_1039770_58.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/5efd77f6-c915-41d9-8832-395783e38af2_calibrate_1039770_58.stderr parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47c81cbd4785c4cede7ee1ef388722ee with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/5b302642-a51e-4d6c-9553-b8518f9f3de4_calibrate_1039674_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/5b302642-a51e-4d6c-9553-b8518f9f3de4_calibrate_1039674_56.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fc727e22c8c77863a46945bd6812df2 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/a032fb44-b37f-48bd-9e4f-01e698bab973_calibrate_1039673_56.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/a032fb44-b37f-48bd-9e4f-01e698bab973_calibrate_1039673_56.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5fddc4a03052d299ac1d569add600a3 with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b5022e83-068c-409f-9650-8374a5b275e0_calibrate_1039673_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/b5022e83-068c-409f-9650-8374a5b275e0_calibrate_1039673_13.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c76f597db4e64b1e47744ddf4f3997 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b5945ab2-6328-4c0b-90e6-d430de64c19e_calibrate_1039769_22.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b5945ab2-6328-4c0b-90e6-d430de64c19e_calibrate_1039769_22.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9225cd40eee332e8d0576a67d066ca45 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/99208013-9889-4f30-bc27-a8353c5dd353_calibrate_1039674_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/99208013-9889-4f30-bc27-a8353c5dd353_calibrate_1039674_44.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25fb71c8b64bd670123c66582b348e1f with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c2485504-1cdc-4491-8c28-becb826a0b05_calibrate_1039674_9.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c2485504-1cdc-4491-8c28-becb826a0b05_calibrate_1039674_9.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc4e222916f89f440cff65af72203c2 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/aef789c7-58d6-4956-9fb4-7e4ac8752234_calibrate_1039768_36.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/aef789c7-58d6-4956-9fb4-7e4ac8752234_calibrate_1039768_36.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4b07a18d147109e1619c7eb07859a69 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/11e88854-a8f0-4ee5-b213-19cdb8573325_calibrate_1039769_26.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/11e88854-a8f0-4ee5-b213-19cdb8573325_calibrate_1039769_26.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3e51f4a8a6bf2dd41c6201d07783960 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14943ec1-4068-47d5-8c8a-055536328d0f_calibrate_1039674_5.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/14943ec1-4068-47d5-8c8a-055536328d0f_calibrate_1039674_5.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 165036fa858d242fbf2e2335396c5c1c with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 2d7611c85c7647168aa753efff011db1 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/99949db4-49e9-49e5-8c9d-bfd27fb04c4c_calibrate_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/99949db4-49e9-49e5-8c9d-bfd27fb04c4c_calibrate_1039769_21.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d91e192f-aae9-4f6c-92b1-638d611c282d_characterizeImage_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/d91e192f-aae9-4f6c-92b1-638d611c282d_characterizeImage_1039769_21.stderr parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b53e27341afc4b6480b1bdc0ea42225b with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/81d6fa47-4b81-4e38-9ecf-785c5d0f47ce_calibrate_1039672_4.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/81d6fa47-4b81-4e38-9ecf-785c5d0f47ce_calibrate_1039672_4.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 635d96103041dbac363edf2a051f20fa with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/beae0f4a-89ec-4c64-8b0f-85041ee3ce04_calibrate_1039769_49.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/beae0f4a-89ec-4c64-8b0f-85041ee3ce04_calibrate_1039769_49.stderr parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1850972153f1cc0d4ba5155a06d91fd6 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bc44f467-b28b-4b83-b817-901a21308c47_calibrate_1039768_51.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bc44f467-b28b-4b83-b817-901a21308c47_calibrate_1039768_51.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30ffab1e743ed8721ba1215d5e902fad with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d79b6233-1a29-4846-a390-e0bcb9a17a3b_calibrate_1039768_57.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d79b6233-1a29-4846-a390-e0bcb9a17a3b_calibrate_1039768_57.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c51504a8f8898e5c818551cfabb0961d with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash fc766b12a68935d9565ebbf6aa0a3332 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8b610b26-c488-4549-a1d1-446bbb85143a_calibrate_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8b610b26-c488-4549-a1d1-446bbb85143a_calibrate_1039770_5.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4171746f-4508-4ff4-8c25-f039d0cea85e_characterizeImage_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/4171746f-4508-4ff4-8c25-f039d0cea85e_characterizeImage_1039770_5.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bf181174cd926ca98e04ea0af19c111 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/7b9cc0a9-13a1-42d5-9b7e-bd8cb86518e0_calibrate_1039768_16.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/7b9cc0a9-13a1-42d5-9b7e-bd8cb86518e0_calibrate_1039768_16.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7016e35d11232a2a5200562a77ff2c3 with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash 2741e0afe8ecbe8894b34da11c6510fd parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fc2b5bab-1ac3-47fa-acb0-71db21e93ee8_calibrate_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fc2b5bab-1ac3-47fa-acb0-71db21e93ee8_calibrate_1039770_54.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a84f70e7-e193-4f07-a521-7ccd66c33e57_characterizeImage_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/a84f70e7-e193-4f07-a521-7ccd66c33e57_characterizeImage_1039770_54.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82ae5edb1b187df355264224a8718b10 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash b9f343f77efcbf231f4f83a39f1acb6d parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e22c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/bb57892e-4e7a-45fe-a08b-89fc3f57b7cd_calibrate_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/bb57892e-4e7a-45fe-a08b-89fc3f57b7cd_calibrate_1039770_38.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d6b4d24c-bd27-4337-b69f-e5dfc21b2ae4_characterizeImage_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/d6b4d24c-bd27-4337-b69f-e5dfc21b2ae4_characterizeImage_1039770_38.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e358ba9419269efcaa49a166ca360764 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/3342d075-0f6b-494e-85a9-91e2758c108c_calibrate_1039672_48.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/3342d075-0f6b-494e-85a9-91e2758c108c_calibrate_1039672_48.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78d7646bdd92ec874312418b4dd897dc with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b5e458c-7f47-45b9-b3a6-5cb642d2f19e_calibrate_1039673_37.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/8b5e458c-7f47-45b9-b3a6-5cb642d2f19e_calibrate_1039673_37.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e77eb43b027702ab5e1803e5061205 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c5dbc536-0f16-4df4-b3d4-4e987060a104_calibrate_1039674_34.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/c5dbc536-0f16-4df4-b3d4-4e987060a104_calibrate_1039674_34.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5799d96372fb82be135644af35b2d11 with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/787c612e-4cc7-47d7-9d82-51b08c9bab17_calibrate_1039768_22.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/787c612e-4cc7-47d7-9d82-51b08c9bab17_calibrate_1039768_22.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8670d16960f2cea0cbf1a90d9e15e0c with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/23e0a77b-6984-4016-9b54-ffbdb00a7e42_calibrate_1039673_16.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/23e0a77b-6984-4016-9b54-ffbdb00a7e42_calibrate_1039673_16.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcc9b13419f6d289fe44dab545d04f28 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4210b8ef-c268-4857-8b68-cd3a78d40390_calibrate_1039769_45.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4210b8ef-c268-4857-8b68-cd3a78d40390_calibrate_1039769_45.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb198a7265f5652cedc42fa82faaaff with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 0582f515599f78ed320eb694195b0dc2 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/da483fe9-1fea-458f-bf73-17a700c7680b_calibrate_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/da483fe9-1fea-458f-bf73-17a700c7680b_calibrate_1039672_24.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/69020a90-21ed-451e-a1ba-c987b0be982b_characterizeImage_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/69020a90-21ed-451e-a1ba-c987b0be982b_characterizeImage_1039672_24.stderr 2024-03-18 04:42:13 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:42:13 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8253ef868f3bb048d43c7f3a34bc342b with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/967e1ba4-9e32-47bc-b82b-2aba5fcd566a_calibrate_1039673_62.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/967e1ba4-9e32-47bc-b82b-2aba5fcd566a_calibrate_1039673_62.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e81a241116743fd2e2974982693abf with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/915a3f0b-63df-4d4a-942f-d70551ee1662_calibrate_1039770_16.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/915a3f0b-63df-4d4a-942f-d70551ee1662_calibrate_1039770_16.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdd06ec11ee1833248913c55a14d501b with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/caa27223-efab-404f-b33b-ceefbe28ec3d_calibrate_1039770_32.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/caa27223-efab-404f-b33b-ceefbe28ec3d_calibrate_1039770_32.stderr parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11d981f93962e34598f28cf8d4d50e15 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c7130d0-d3ba-428f-9899-602f23b0d657_calibrate_1039673_9.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3c7130d0-d3ba-428f-9899-602f23b0d657_calibrate_1039673_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:13 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339dd25cb5669927f9d92c09f3cf01df with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f89b00a1-82b8-404e-abd8-797b6d50017d_calibrate_1039770_46.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f89b00a1-82b8-404e-abd8-797b6d50017d_calibrate_1039770_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:14 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1464be4be0014f31a02d5800fb019114 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/196428fc-a145-48a3-b9bf-80cacb3733b5_calibrate_1039770_3.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/196428fc-a145-48a3-b9bf-80cacb3733b5_calibrate_1039770_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:14 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac74010eb3311ecec116580fe28391a2 with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5cb0bc4-3b02-4d58-883c-a4e52d1a472b_calibrate_1039768_17.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5cb0bc4-3b02-4d58-883c-a4e52d1a472b_calibrate_1039768_17.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4d42560968f61af1ada27d7b56af20d with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7a9b54d9-e024-4c7a-9e21-c10815e3b61f_calibrate_1039769_59.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7a9b54d9-e024-4c7a-9e21-c10815e3b61f_calibrate_1039769_59.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d3ad724b4b8346e3500eb5ee74bfdcd with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/956c5d3b-2054-4d7a-b7f2-b041fcfcc8b2_calibrate_1039769_39.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/956c5d3b-2054-4d7a-b7f2-b041fcfcc8b2_calibrate_1039769_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:15 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:15 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38c05b2815df0d8e63dbc2df5227761b with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11267445-0e85-4f3c-8d9d-9bc73d55d930_calibrate_1039770_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11267445-0e85-4f3c-8d9d-9bc73d55d930_calibrate_1039770_45.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33b58fa52f670ebfa81f76393e416e2d with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash eec15d997f5cbd03f7e58242768ab54f parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e383ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/86f22a33-6f1a-4f86-ad05-9da3ed340920_calibrate_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/86f22a33-6f1a-4f86-ad05-9da3ed340920_calibrate_1039770_59.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b92e76c0-807d-4bce-81e8-636a8585a8e8_characterizeImage_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/b92e76c0-807d-4bce-81e8-636a8585a8e8_characterizeImage_1039770_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:16 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ffa25e402fa29a495182d2fbe1ebea6 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3e01ffa-943d-4da6-a0da-ca1b3b32522f_calibrate_1039673_14.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e3e01ffa-943d-4da6-a0da-ca1b3b32522f_calibrate_1039673_14.stderr parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c3f92cb6d754589efa569e7c64178e1 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/79e98967-02dc-49b7-a09f-11a10a987ab6_calibrate_1039769_54.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/79e98967-02dc-49b7-a09f-11a10a987ab6_calibrate_1039769_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:16 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb63e5c84fe6d41610643b64503ef58e with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3da54457-054f-45ca-b420-0db7f6d650c7_calibrate_1039769_3.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3da54457-054f-45ca-b420-0db7f6d650c7_calibrate_1039769_3.stderr parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be4b2be24b1baea46ac27100ed989fee with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f5550704-a5c9-4959-ac9f-559210c13290_calibrate_1039768_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f5550704-a5c9-4959-ac9f-559210c13290_calibrate_1039768_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:17 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1066c658d39a8ef67e07212e422f5c with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6b28b54a-df9b-4efa-acca-590d5b421300_calibrate_1039770_37.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6b28b54a-df9b-4efa-acca-590d5b421300_calibrate_1039770_37.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e47f2be449cd68f6ff044d50a7db4a07 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/2c098d4f-2f6f-49f7-a6ec-875dae1d72b6_calibrate_1039672_44.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/2c098d4f-2f6f-49f7-a6ec-875dae1d72b6_calibrate_1039672_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:17 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3019ae2f67fb7f709603cf39034c9e with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1712f189-5092-4bf8-a31c-d5d4de533ca7_calibrate_1039673_26.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1712f189-5092-4bf8-a31c-d5d4de533ca7_calibrate_1039673_26.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3e06d6598ea3ecb0196b83e8441dfe5 with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0d54a077-66a5-477c-89f0-d1f2a4ab3250_calibrate_1039672_37.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/0d54a077-66a5-477c-89f0-d1f2a4ab3250_calibrate_1039672_37.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7dc73b76f2cffacb7e0dbd4ddcb6963 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5609b5bc-cda4-45ca-8c49-2ee7d9acb754_calibrate_1039769_50.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5609b5bc-cda4-45ca-8c49-2ee7d9acb754_calibrate_1039769_50.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 798176a05a1e60b03efb16c5360186a1 with result from task 265 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:17 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 2cacd1371db950e88038570b83f98d93 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f93a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/162603c2-32ca-4d78-8617-8f237b00060c_calibrate_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/162603c2-32ca-4d78-8617-8f237b00060c_calibrate_1039672_62.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92b5740b-4c21-49bb-9b70-ae17659e09fe_characterizeImage_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/92b5740b-4c21-49bb-9b70-ae17659e09fe_characterizeImage_1039672_62.stderr parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7853275b2d1849d0bf546fab35a0da97 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a4281c0b-9ccd-4d1e-8d2c-93dcedeb72d8_calibrate_1039768_41.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/a4281c0b-9ccd-4d1e-8d2c-93dcedeb72d8_calibrate_1039768_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:18 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d335742e295d11c1a9494000cdc82284 with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d456d604-ebc1-45d6-838c-c0da798eb488_calibrate_1039672_34.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/d456d604-ebc1-45d6-838c-c0da798eb488_calibrate_1039672_34.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e172475fa90a55e7fcf83c415c3a69d with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/a26c7cd2-1f5c-4247-8584-1f0a2b7baf28_calibrate_1039769_5.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/a26c7cd2-1f5c-4247-8584-1f0a2b7baf28_calibrate_1039769_5.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4238661fee35ce1943733f42806ed7e1 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d493e19-afc0-49bb-a624-af8d0b1ac5af_calibrate_1039672_55.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/7d493e19-afc0-49bb-a624-af8d0b1ac5af_calibrate_1039672_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:18 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f04da7ef8174fade9894c303877a36f2 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e2abc373-d8f0-4225-a095-505311163c0e_calibrate_1039769_27.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e2abc373-d8f0-4225-a095-505311163c0e_calibrate_1039769_27.stderr parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2afdb30ede318604a38562bcf5cc8090 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4eda5277-8d02-4589-bca1-8fd00aaecff6_calibrate_1039674_14.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4eda5277-8d02-4589-bca1-8fd00aaecff6_calibrate_1039674_14.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a14849c38b076750d4a412921df234 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5eace840-91bb-419a-86a5-f9e3efe2068a_calibrate_1039768_62.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5eace840-91bb-419a-86a5-f9e3efe2068a_calibrate_1039768_62.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa623ccd543a54df48e59fe41d783a2f with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/61f10922-a549-43b4-a177-00c56e87a426_calibrate_1039769_42.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/61f10922-a549-43b4-a177-00c56e87a426_calibrate_1039769_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:19 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c04c1dfaad5f662e3f72f53f24d2c28 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e784b519-2f79-4707-9c69-a3b3b9376001_calibrate_1039673_54.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/e784b519-2f79-4707-9c69-a3b3b9376001_calibrate_1039673_54.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3f0b3f9a439d13565bf0e0c04207fc8 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/332f0ce0-ccf4-4622-ae70-fb14df63aec2_calibrate_1039770_25.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/332f0ce0-ccf4-4622-ae70-fb14df63aec2_calibrate_1039770_25.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 383325fcecc36f4c4a1b80e73ee66bbf with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ce5204a3-4db0-4c9e-9316-a3c7d7a9f198_calibrate_1039768_39.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ce5204a3-4db0-4c9e-9316-a3c7d7a9f198_calibrate_1039768_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:20 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c0d28e240d762e4fd844a82de4776d with result from task 395 parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:20 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/b6d127ab-5b87-40a9-9fc5-876c3890f2bb_calibrate_1039768_33.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/b6d127ab-5b87-40a9-9fc5-876c3890f2bb_calibrate_1039768_33.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cc996a245b26195c9608711fd1bf8d1 with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3d373a3c-3bb6-48f8-9660-8c49db0eb23a_calibrate_1039768_21.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/3d373a3c-3bb6-48f8-9660-8c49db0eb23a_calibrate_1039768_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:21 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada818e16c768e7619ba989c7fcab709 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/04d3d496-ba2c-4655-bb44-474dd663d56d_calibrate_1039673_34.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/04d3d496-ba2c-4655-bb44-474dd663d56d_calibrate_1039673_34.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a837c31cb6b4c0860e89de66bb85396d with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash f76d07a2e35cf67bc7d0eb21be5ffac1 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4516a2de-5564-4369-bb86-8245ed97130b_calibrate_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4516a2de-5564-4369-bb86-8245ed97130b_calibrate_1039769_9.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/98d0857a-07a6-40c1-9236-d239e3da8cd6_characterizeImage_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/98d0857a-07a6-40c1-9236-d239e3da8cd6_characterizeImage_1039769_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:21 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e859b41810cb56f02cb1ed7d8ad45d3 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash fd7942ec9832917b598402e332ea5bca parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/53da13c7-cd3c-4cbc-bd6d-cc2e45312746_calibrate_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/53da13c7-cd3c-4cbc-bd6d-cc2e45312746_calibrate_1039769_44.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/2becc26d-59ad-454b-a1ba-80a1967b7cba_characterizeImage_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/2becc26d-59ad-454b-a1ba-80a1967b7cba_characterizeImage_1039769_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:22 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9851dbf06a4ef8d5501ff2b595c5ac6 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ed846e42-e7dc-490b-bb5b-3d2c883938aa_calibrate_1039770_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ed846e42-e7dc-490b-bb5b-3d2c883938aa_calibrate_1039770_40.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5faa32b382c6cbefc616d747fe443e with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/2ea26240-4f08-467a-a7d1-bc2e95e7e8a9_calibrate_1039769_57.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/2ea26240-4f08-467a-a7d1-bc2e95e7e8a9_calibrate_1039769_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:22 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bd984a8e816a0c0255400560fba3903 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4d2ba30f-38d2-47c9-8cf8-5eccc5b60afa_calibrate_1039768_10.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4d2ba30f-38d2-47c9-8cf8-5eccc5b60afa_calibrate_1039768_10.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09c4dc14d3b646171fd41c2951f9cf81 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/882fb411-3ee0-486e-9ac9-ada124a309f1_calibrate_1039770_41.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/882fb411-3ee0-486e-9ac9-ada124a309f1_calibrate_1039770_41.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d99691f52d05602a9463b0686c8e09c with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/32101117-f28e-4e7d-9ab3-2fe1c49a566f_calibrate_1039769_35.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/32101117-f28e-4e7d-9ab3-2fe1c49a566f_calibrate_1039769_35.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba35adc8b0f8b40471cd3a987ebe842f with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5388ac1e-3a49-4e01-b222-a3ab5c526139_calibrate_1039769_13.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5388ac1e-3a49-4e01-b222-a3ab5c526139_calibrate_1039769_13.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e67d8fc8d3f5fcca339bfbce116554d0 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/95406503-fd2a-45a6-a443-2a2261048888_calibrate_1039768_4.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/95406503-fd2a-45a6-a443-2a2261048888_calibrate_1039768_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:23 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdcc3fc9b9c3ad98bd0ca8cbc27831ac with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1accd337-b526-45cf-83e6-7c291db89538_calibrate_1039673_52.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1accd337-b526-45cf-83e6-7c291db89538_calibrate_1039673_52.stderr parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08d0c4912269c9f37007207ccb0a2bf4 with result from task 1004 parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/36342f51-d29f-4876-81cc-27a69bdc3d5f_calibrate_1039770_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/36342f51-d29f-4876-81cc-27a69bdc3d5f_calibrate_1039770_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:23 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67af9cfd2ba260fa67eccb87a26b46b0 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ee695afb-e12b-42f7-a7d0-df7acb53e61d_calibrate_1039770_42.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ee695afb-e12b-42f7-a7d0-df7acb53e61d_calibrate_1039770_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:24 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61b6552bda0dca02d330c3e1c8ee4c66 with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/48c20a88-3966-478a-8460-44c5474d85de_calibrate_1039770_31.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/48c20a88-3966-478a-8460-44c5474d85de_calibrate_1039770_31.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38249480e96d6722c7f17e1ee027aabb with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e18548a2-6c54-4ebc-9a8b-d721fd64d9e5_calibrate_1039769_10.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/e18548a2-6c54-4ebc-9a8b-d721fd64d9e5_calibrate_1039769_10.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e825a32976253ec5e662ad8f292ee195 with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/83deedfc-386e-4b43-a5c2-6de1d8689ab8_calibrate_1039769_1.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/83deedfc-386e-4b43-a5c2-6de1d8689ab8_calibrate_1039769_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:24 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c5efb7a8768fc039d57646f64021c2c with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1f7440c8-aba1-4602-bba7-b7ca4863a7cf_calibrate_1039673_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/1f7440c8-aba1-4602-bba7-b7ca4863a7cf_calibrate_1039673_3.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c73fe8eba524d833dec4ebd3731a00 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash d1752d64ac5d1e18a7af07ea3d8083ae parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2fde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c7c914cb-1fad-4773-af11-3cbd7384bd0b_calibrate_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c7c914cb-1fad-4773-af11-3cbd7384bd0b_calibrate_1039770_44.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/731773ee-ab9d-4225-b462-a2b205435bc1_characterizeImage_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/731773ee-ab9d-4225-b462-a2b205435bc1_characterizeImage_1039770_44.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15266c5be337cc2589417cd55d81ac5b with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/9e397aa9-1fca-4efd-a102-7e4bb2637011_calibrate_1039769_4.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/9e397aa9-1fca-4efd-a102-7e4bb2637011_calibrate_1039769_4.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b3c21d1e2c65f999732c316148a61d with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/43d5e0c2-4cc2-484e-a05d-c7949942dac1_calibrate_1039770_23.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/43d5e0c2-4cc2-484e-a05d-c7949942dac1_calibrate_1039770_23.stderr parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10ff8f68167b35c2dccdac8b3b7047ff with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/92ba83b9-9fa4-41ae-8c41-3a523a79c490_calibrate_1039674_39.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/92ba83b9-9fa4-41ae-8c41-3a523a79c490_calibrate_1039674_39.stderr parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc4583b7b4acc46224fcd7eb1726dbc3 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0da58550-e457-4f5a-a5f4-fd7798ed58b0_calibrate_1039769_12.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/0da58550-e457-4f5a-a5f4-fd7798ed58b0_calibrate_1039769_12.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6e80ae3691e147dfab084f9d1b92cc6 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 31ba8df40e5d4ecde061e90a7efc8483 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e35d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b473e5ba-7bd0-4671-9c7b-b85ccbe3cff0_calibrate_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b473e5ba-7bd0-4671-9c7b-b85ccbe3cff0_calibrate_1039769_62.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/60010f75-8990-4cb1-aa49-0f701ac6ba91_characterizeImage_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/60010f75-8990-4cb1-aa49-0f701ac6ba91_characterizeImage_1039769_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:25 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f045fc907fe0afd68e1aeb2045f8dad3 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash f4454d67cc9fd962031c9fc76eb50eba parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e14c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/44ebc5fc-0733-4540-8f58-0268fa2bf1f3_calibrate_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/44ebc5fc-0733-4540-8f58-0268fa2bf1f3_calibrate_1039672_47.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/064c05fc-3755-4b11-91e3-9741e17db24b_characterizeImage_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/064c05fc-3755-4b11-91e3-9741e17db24b_characterizeImage_1039672_47.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06aaa1cf442b4d7b3824e3cfc663f783 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/903038a0-9170-4798-8124-c06246dd3339_calibrate_1039770_7.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/903038a0-9170-4798-8124-c06246dd3339_calibrate_1039770_7.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f0b090af94d3af2adee5e8d4d4133e3 with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f102153c-9c4b-4888-99b7-5fe2212f509c_calibrate_1039768_49.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f102153c-9c4b-4888-99b7-5fe2212f509c_calibrate_1039768_49.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7320a61de770f60874a854058ef0a3d with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f62e036a-d033-456a-bca0-420b61e2ad83_calibrate_1039672_9.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f62e036a-d033-456a-bca0-420b61e2ad83_calibrate_1039672_9.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e28e707529d2b8d824ec61ff815c1c48 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bd22e264-964e-4b50-96e5-d6254a7af605_calibrate_1039769_52.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bd22e264-964e-4b50-96e5-d6254a7af605_calibrate_1039769_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:25 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c8aaed0d9b6feccee75ad2451ad580 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/450010c5-5b5f-417d-93ae-c916b151be8e_calibrate_1039768_12.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/450010c5-5b5f-417d-93ae-c916b151be8e_calibrate_1039768_12.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c56c8e2c8819e98753711ac650ad62e7 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ad152488-4319-485b-99b6-3b3750a0f1b8_calibrate_1039770_34.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ad152488-4319-485b-99b6-3b3750a0f1b8_calibrate_1039770_34.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe139607cba30c2738a50919d2ad5cc with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5950737b-6473-4a2a-a39c-613dadcee13f_calibrate_1039769_34.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5950737b-6473-4a2a-a39c-613dadcee13f_calibrate_1039769_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:26 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aab5f6876c7aac4f6b5b5c451694354e with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4e8fb25c-c694-4360-87ef-82a034236537_calibrate_1039770_53.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4e8fb25c-c694-4360-87ef-82a034236537_calibrate_1039770_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:26 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6432376280d99f79bc09348240d5156 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11ec19b4-e9b4-43e4-a1d0-6f58307fb089_calibrate_1039770_24.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/11ec19b4-e9b4-43e4-a1d0-6f58307fb089_calibrate_1039770_24.stderr parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0a4bf7ba69dbd718666355f2fdfc4e1 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3a3b223d-76b9-421b-b5ca-564c3221516a_calibrate_1039769_8.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/3a3b223d-76b9-421b-b5ca-564c3221516a_calibrate_1039769_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:27 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2438d343de4ebc9d1b71ab4ce1814817 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ce3bbb42-050b-4dab-9db4-3407080ca5c7_calibrate_1039770_26.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/ce3bbb42-050b-4dab-9db4-3407080ca5c7_calibrate_1039770_26.stderr parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 894133cd79eb0823190d5bb8416a1bf4 with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7887f4c1-5da2-4995-8a7d-a6bd07d56793_calibrate_1039673_47.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7887f4c1-5da2-4995-8a7d-a6bd07d56793_calibrate_1039673_47.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85244575c0f5c474b4e1f98f77cb1ca9 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/95553e53-03e8-42e5-bef1-34a241fa103c_calibrate_1039672_26.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/95553e53-03e8-42e5-bef1-34a241fa103c_calibrate_1039672_26.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe068d655f9e2b9c548c44067eb9d052 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/10b0040f-2136-495f-a52d-baa4f483b252_calibrate_1039768_24.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/10b0040f-2136-495f-a52d-baa4f483b252_calibrate_1039768_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:30 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1eb8380dd6e1160ffcc943a8b84a3c60 with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c71c3bd0-f352-42a0-b17c-ffbdc04aee33_calibrate_1039769_33.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c71c3bd0-f352-42a0-b17c-ffbdc04aee33_calibrate_1039769_33.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 316749764e5a8b0aa5bd2d8db95c10fd with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/85531c43-e07a-4f5f-a720-cca0172ab61a_calibrate_1039673_5.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/85531c43-e07a-4f5f-a720-cca0172ab61a_calibrate_1039673_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:30 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3da3053a83fd2e587e0a1443ef55e343 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/382fabb3-2993-45a0-b00c-52d825524327_calibrate_1039674_50.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/382fabb3-2993-45a0-b00c-52d825524327_calibrate_1039674_50.stderr parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed91303e2ccdfb86480b702962ffe410 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash e5ecd4db13fbc6d58bc9632137cf867a parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3baac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/73e3d084-7e87-4ac6-8a4b-4f6929a4cc35_calibrate_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/73e3d084-7e87-4ac6-8a4b-4f6929a4cc35_calibrate_1039673_15.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1c547a-9edf-41ab-be93-a0a63400248b_characterizeImage_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/7a1c547a-9edf-41ab-be93-a0a63400248b_characterizeImage_1039673_15.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 984772eb2dab35d9859576acf0a85435 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a81c64fb-e740-4bcd-bd61-dd2a2449a005_calibrate_1039674_33.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/a81c64fb-e740-4bcd-bd61-dd2a2449a005_calibrate_1039674_33.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce76ecfa5a04ce5422c0e9dea11f7985 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4c72a321-0c09-42fe-8024-52592f3c9ded_calibrate_1039770_51.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/4c72a321-0c09-42fe-8024-52592f3c9ded_calibrate_1039770_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:32 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26fb4febef2d147441f40c984f4a9bda with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d4d51fe6-d791-43d9-acc1-5dce218abc03_calibrate_1039770_27.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/d4d51fe6-d791-43d9-acc1-5dce218abc03_calibrate_1039770_27.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2710caab0ed11e221ee40a686ab2eb3 with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b68e3803-4ce2-4e6c-a25c-ee495a2a5a2c_calibrate_1039769_51.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b68e3803-4ce2-4e6c-a25c-ee495a2a5a2c_calibrate_1039769_51.stderr parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f25ac29de606652a9d4fb0a737c45bb with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8740d17a-6fa3-42ad-9bc7-ed687f409a49_calibrate_1039672_5.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/8740d17a-6fa3-42ad-9bc7-ed687f409a49_calibrate_1039672_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:33 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63fa7178c7052eed8a761dc28f6cb273 with result from task 1022 parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f551c325-7ef2-482f-9ea0-1d419a008276_calibrate_1039672_41.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f551c325-7ef2-482f-9ea0-1d419a008276_calibrate_1039672_41.stderr parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b1d0aeb5a704d990bbf9b58a7f5f53 with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/290e4bed-d6c6-49e5-a515-7e444fdfc2d1_calibrate_1039672_12.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/290e4bed-d6c6-49e5-a515-7e444fdfc2d1_calibrate_1039672_12.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 254a27e4572ecb51515228287951c2b1 with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash fc21e57ce6aad927404d3213fa87f248 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/92f1478d-7f5c-458c-9ce7-0458006e1238_calibrate_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/92f1478d-7f5c-458c-9ce7-0458006e1238_calibrate_1039768_35.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ae14ec0f-4d1b-4b4d-9b07-b8742109ee42_characterizeImage_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/ae14ec0f-4d1b-4b4d-9b07-b8742109ee42_characterizeImage_1039768_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:34 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:35 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbb812a4b03b31e58bde236b55e1f609 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5a6d01cf-78a4-40ef-8649-5bb4d9dc8405_calibrate_1039769_38.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5a6d01cf-78a4-40ef-8649-5bb4d9dc8405_calibrate_1039769_38.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f672591f5b8a3f35bbdb619aa77766c0 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f921f64e-078c-45e1-b003-60ecf94d559b_calibrate_1039672_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/f921f64e-078c-45e1-b003-60ecf94d559b_calibrate_1039672_30.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63b0deaa0f560e0b145827fd1e48e7c7 with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c5a3fcea-2052-4d64-9894-e181636af026_calibrate_1039769_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c5a3fcea-2052-4d64-9894-e181636af026_calibrate_1039769_25.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 653b3c81c9db2cf20dded2a804cdbbed with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3eef43cc-bc8f-4d54-a703-a4f93f5c6ebb_calibrate_1039770_39.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3eef43cc-bc8f-4d54-a703-a4f93f5c6ebb_calibrate_1039770_39.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 100973a542f39532f049a495ace26b2d with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b3358c93-e8a2-4e4e-8fcc-adda62d2cab9_calibrate_1039770_49.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/b3358c93-e8a2-4e4e-8fcc-adda62d2cab9_calibrate_1039770_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:36 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e7a38408fbf532a3a245a362a2a3f5d with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea4d2830-a950-43e0-ba26-82396dafd714_calibrate_1039769_6.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea4d2830-a950-43e0-ba26-82396dafd714_calibrate_1039769_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a010efbd2195350471cebd74ae392740 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/20e0518c-d2d8-4820-b264-897430f00754_calibrate_1039768_48.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/20e0518c-d2d8-4820-b264-897430f00754_calibrate_1039768_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1362c9eff27888318228eef71116de3 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c03eea5f-df62-437f-be2d-8e9b5e268a31_calibrate_1039769_20.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c03eea5f-df62-437f-be2d-8e9b5e268a31_calibrate_1039769_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc6e9e432a5117d68178010445922005 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/abce862a-9a82-4af2-be99-ea87a2f25898_calibrate_1039769_41.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/abce862a-9a82-4af2-be99-ea87a2f25898_calibrate_1039769_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec99a969d389ebdf821221036f1b606a with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/26a2ca98-0899-42df-9d1b-3a69d8092a7a_calibrate_1039674_62.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/26a2ca98-0899-42df-9d1b-3a69d8092a7a_calibrate_1039674_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:37 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db8e98995ea9b55957b6fd69cc9a6610 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/1602b8ba-cf64-47fb-99f8-00f1d982227c_calibrate_1039769_11.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/1602b8ba-cf64-47fb-99f8-00f1d982227c_calibrate_1039769_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4530f9b2e72293410a7fe708b61f118a with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6974ff9a-1a0f-4fb9-9fd5-f186b5274e7c_calibrate_1039674_25.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/6974ff9a-1a0f-4fb9-9fd5-f186b5274e7c_calibrate_1039674_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:38 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066734d917e661afdb0d2a0db08a6beb with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/568d7614-1d4d-4981-b2ec-248e6c41f584_calibrate_1039672_60.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/568d7614-1d4d-4981-b2ec-248e6c41f584_calibrate_1039672_60.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1149a0841d7d8030b948a168851480fe with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d2255967-b238-4c6c-be0f-418688ba182e_calibrate_1039674_19.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/d2255967-b238-4c6c-be0f-418688ba182e_calibrate_1039674_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:39 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f79338604acff9993c38193841ddb56e with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a765cd63-de86-4e50-b28e-3e2fdabf098f_calibrate_1039672_54.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/a765cd63-de86-4e50-b28e-3e2fdabf098f_calibrate_1039672_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59dde85e1ec413f367a2b6261a9a7019 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 4cc919eda1286cb3083d39753b8c4ea2 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2787c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f2e26fe4-8570-42e3-87db-408deb5de67f_characterizeImage_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/f2e26fe4-8570-42e3-87db-408deb5de67f_characterizeImage_1039769_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:40 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:41 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05057aeffe36deb0174dac1574dbab5f with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4958bdfd-4cc4-4535-adae-eb44d603a6c4_calibrate_1039674_15.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/4958bdfd-4cc4-4535-adae-eb44d603a6c4_calibrate_1039674_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:42 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dffaed758690674eb4f7a5181f06be2 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/baedfe59-4048-484e-b44b-c2a09da9d495_calibrate_1039770_57.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/baedfe59-4048-484e-b44b-c2a09da9d495_calibrate_1039770_57.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80027248b18e854dbf96717f36bc4e0e with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8986bb68-d012-4d24-9626-88c17dcf3780_calibrate_1039770_14.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8986bb68-d012-4d24-9626-88c17dcf3780_calibrate_1039770_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92f6449cdc90c0f65de1c3dc2875e151 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6de0535d-dcf1-452d-afd4-6734c58db20e_calibrate_1039768_58.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6de0535d-dcf1-452d-afd4-6734c58db20e_calibrate_1039768_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:43 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a151b3fe203e3836068e33e24aa19ef7 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/eb11b2ac-498b-4281-9349-d80f52d2ba6e_calibrate_1039770_19.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/eb11b2ac-498b-4281-9349-d80f52d2ba6e_calibrate_1039770_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e75749e9b3e30a692237eb525870abd with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c57107db-ddde-4d87-a65f-d24f223ee2de_calibrate_1039770_48.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c57107db-ddde-4d87-a65f-d24f223ee2de_calibrate_1039770_48.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd9e98f3ff236653477a4dc15a5510e7 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/27142b82-9401-46af-b8db-f3d4bd1d9322_calibrate_1039769_19.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/27142b82-9401-46af-b8db-f3d4bd1d9322_calibrate_1039769_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be5591ff71eef427c2853a3b20a1ee3b with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f204060d-58aa-4d99-a3d4-5e1e0b9fbdc5_calibrate_1039674_26.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/f204060d-58aa-4d99-a3d4-5e1e0b9fbdc5_calibrate_1039674_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 77 parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4f6e320dba8859fe5013a1a51e0622 with result from task 851 parsl.dataflow.strategy DEBUG: Executor multi has 77 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/862ed1af-e35f-462f-8131-3d391734d74b_calibrate_1039672_14.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/862ed1af-e35f-462f-8131-3d391734d74b_calibrate_1039672_14.stderr 2024-03-18 04:42:44 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b02b1a891401efe03b77d80cfa8cf8a with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9b843319-4a82-4b37-a419-e82ecae9b49b_calibrate_1039768_5.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9b843319-4a82-4b37-a419-e82ecae9b49b_calibrate_1039768_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:45 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df53dd7e35acac968d31d39ab97fc5e3 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f7c3926b-e68a-4caf-8aaa-737d501ef78f_calibrate_1039768_54.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/f7c3926b-e68a-4caf-8aaa-737d501ef78f_calibrate_1039768_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41067d9b489ac460c37bf4c6bab39d7 with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/48fbef5c-67b5-4d0d-96f3-0ab4108208a6_calibrate_1039673_59.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/48fbef5c-67b5-4d0d-96f3-0ab4108208a6_calibrate_1039673_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a43bac5b7ac56f94495206db7956043c with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cd28e715-9072-4a81-8ee6-e03541c4f204_calibrate_1039769_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cd28e715-9072-4a81-8ee6-e03541c4f204_calibrate_1039769_17.stderr parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 155206f5b72c69a9f75c38fac49c81a3 with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/df5fd3a9-9bbc-4094-850e-fd56ee24416e_calibrate_1039674_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/df5fd3a9-9bbc-4094-850e-fd56ee24416e_calibrate_1039674_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:46 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03c133a745ca36eb1bf28ca1128b7c47 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec5b362b-2745-4153-ad3e-22939ab07631_calibrate_1039768_47.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec5b362b-2745-4153-ad3e-22939ab07631_calibrate_1039768_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf01f916ee9e8ae7d23aa2c870687f46 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c6c40178-9d72-4a85-8a2b-055069f81c9c_calibrate_1039770_50.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c6c40178-9d72-4a85-8a2b-055069f81c9c_calibrate_1039770_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ab6b11c3c7482efd0486a7acc883a88 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/0c9e96ac-142e-48c0-9084-9a11850f2ec7_calibrate_1039770_55.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/0c9e96ac-142e-48c0-9084-9a11850f2ec7_calibrate_1039770_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 574b15a420b7463725ea05060925dcd6 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/dc885268-1a4c-4f56-a9f0-7a01253c6d2d_calibrate_1039672_23.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/dc885268-1a4c-4f56-a9f0-7a01253c6d2d_calibrate_1039672_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:47 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52b63679f487bde4ade59afdf3038698 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4b12ce15-ee5d-4a68-bc10-7a688f3d460f_calibrate_1039768_27.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/4b12ce15-ee5d-4a68-bc10-7a688f3d460f_calibrate_1039768_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5f4abb0c7dadef1086ebbd8ef88c44f with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c0e5608b-fbf6-434f-ab6f-359099d4e7b7_calibrate_1039769_56.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/c0e5608b-fbf6-434f-ab6f-359099d4e7b7_calibrate_1039769_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:48 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f0268e30e19957ec01c694efae8f92b with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fab5e3bc-4efb-4f8c-bc16-d6e4cf3dc483_calibrate_1039674_58.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/fab5e3bc-4efb-4f8c-bc16-d6e4cf3dc483_calibrate_1039674_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:49 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:50 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0970382419636ee77f01cb617cbeeae with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea6d6f27-f247-4aad-a87c-6d25f39bfe05_calibrate_1039769_40.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea6d6f27-f247-4aad-a87c-6d25f39bfe05_calibrate_1039769_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6759e5fdcfbf8a9d73ed8f9c6967e076 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/cdcea881-1f38-48d1-ba46-34be9ca08c6e_calibrate_1039768_7.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/cdcea881-1f38-48d1-ba46-34be9ca08c6e_calibrate_1039768_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:51 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 392a172e2a04254168cb1e1957a25c29 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/991e745d-0f22-4d17-9f68-a8f1214ec459_calibrate_1039769_55.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/991e745d-0f22-4d17-9f68-a8f1214ec459_calibrate_1039769_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:52 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 498a6f1188a3578745135c1bb9a913d5 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/df17abff-7724-4b4f-b064-7dcc39fd25d6_calibrate_1039769_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/df17abff-7724-4b4f-b064-7dcc39fd25d6_calibrate_1039769_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2dd4a99ad20326682d8fa00a9b91e81 with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash ed3c314f097cd3ca1c26ac7bc36f27a8 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/8d12935f-9b67-4008-b81a-32354a87f827_characterizeImage_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/8d12935f-9b67-4008-b81a-32354a87f827_characterizeImage_1039770_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:53 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd3e35bf8e504c9863186420eb6702bd with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7dad39df-89ef-48c7-86ee-66a83cd03880_calibrate_1039673_41.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/7dad39df-89ef-48c7-86ee-66a83cd03880_calibrate_1039673_41.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4417f56fddfce2f19117f05ffb1fe3ab with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5368c92-8c3f-4e33-b009-caaeefb93a44_calibrate_1039768_56.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5368c92-8c3f-4e33-b009-caaeefb93a44_calibrate_1039768_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31df165789bc7481827cc383b376a3f2 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c62b57a7-1561-4226-beb5-769588482f24_calibrate_1039770_60.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c62b57a7-1561-4226-beb5-769588482f24_calibrate_1039770_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1276b3f9bd54415a04a8c80fef95c330 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash 2af1da446146f9bd3dce47a9f0481ef8 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2ff600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2036e7bd-29a3-4a1a-bd76-47299914de55_calibrate_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2036e7bd-29a3-4a1a-bd76-47299914de55_calibrate_1039770_62.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce3fae6a-f1b0-493c-ba4f-992aebbc7a9e_characterizeImage_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/ce3fae6a-f1b0-493c-ba4f-992aebbc7a9e_characterizeImage_1039770_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e59c18c3d705e376aef7917418545257 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/061f6a16-46df-4292-81f7-1959428df6dd_calibrate_1039768_28.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/061f6a16-46df-4292-81f7-1959428df6dd_calibrate_1039768_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:54 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93ecf651efc1ab1ff0ab6604afd75d18 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/46668289-8208-440b-84b2-9448fb18164f_calibrate_1039770_56.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/46668289-8208-440b-84b2-9448fb18164f_calibrate_1039770_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:55 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 153a79983aaf9262448ce4b06cfe19b9 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d2255a33-7d4e-4e35-9fdf-4b2ab1ed3230_calibrate_1039769_16.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d2255a33-7d4e-4e35-9fdf-4b2ab1ed3230_calibrate_1039769_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f15ba3bd010341f95c49fa971db3807 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/411767ec-c616-4251-bc14-90a16ca7a5ce_calibrate_1039770_43.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/411767ec-c616-4251-bc14-90a16ca7a5ce_calibrate_1039770_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:56 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46f304273ef49ffb0cf2c6640b61724 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5d290c0b-81b1-479e-88e3-9c0d6ce9de18_calibrate_1039768_20.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/5d290c0b-81b1-479e-88e3-9c0d6ce9de18_calibrate_1039768_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:57 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:42:57 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 52 parsl.dataflow.strategy DEBUG: Executor multi has 52 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 52 parsl.dataflow.strategy DEBUG: Executor multi has 52 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ea5c89921f1ea2f8e4cc030d890bed with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/9febbf66-1f7b-4760-a359-fba5614aff76_calibrate_1039673_58.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/9febbf66-1f7b-4760-a359-fba5614aff76_calibrate_1039673_58.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e44b59f9215ffd7d81b62ee6941e0d with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea7a78a7-c893-4f94-88c6-e6bed2f21ea9_calibrate_1039769_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/ea7a78a7-c893-4f94-88c6-e6bed2f21ea9_calibrate_1039769_29.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b8e9a1e7305410724e0bfbcf091f5d3 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/50f31899-c526-44a0-9b33-63157a3988fa_calibrate_1039768_30.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/50f31899-c526-44a0-9b33-63157a3988fa_calibrate_1039768_30.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d2b73227af23b5270ea33f100d5e03b with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3175324b-5e53-436c-bff8-5475e894d297_calibrate_1039673_12.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/3175324b-5e53-436c-bff8-5475e894d297_calibrate_1039673_12.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbf24b81c4cc54023367a0cee72622ee with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/88e96da5-b09f-42e9-95ee-84c9d2d7a859_calibrate_1039769_30.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/88e96da5-b09f-42e9-95ee-84c9d2d7a859_calibrate_1039769_30.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 156c9673e879eb45b5a76cb4db494ea3 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5fce390-3fa4-4b4f-8e0c-d4c2316b8aba_calibrate_1039768_38.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/e5fce390-3fa4-4b4f-8e0c-d4c2316b8aba_calibrate_1039768_38.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3657ee8e888c1cd93cdb1ee52faeefd5 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/19a0d367-c72b-4feb-99f7-2cc1b7242548_calibrate_1039768_40.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/19a0d367-c72b-4feb-99f7-2cc1b7242548_calibrate_1039768_40.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cacd1371db950e88038570b83f98d93 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/162603c2-32ca-4d78-8617-8f237b00060c_calibrate_1039672_62.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/162603c2-32ca-4d78-8617-8f237b00060c_calibrate_1039672_62.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6337704c58d5cad6c5c8d502e65d138 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8585b4c3-d6cf-45c8-9dfc-99539e2d7e6f_calibrate_1039770_28.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8585b4c3-d6cf-45c8-9dfc-99539e2d7e6f_calibrate_1039770_28.stderr parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cb3ed18d47814f863b7625738674dba with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bf1ce586-c861-41b9-9913-23986504d43e_calibrate_1039769_18.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/bf1ce586-c861-41b9-9913-23986504d43e_calibrate_1039769_18.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60202f1e1f3869774b31f5384846b0e6 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6a1371b3-80c2-4ca9-960a-c54ef4f4cec7_calibrate_1039770_6.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/6a1371b3-80c2-4ca9-960a-c54ef4f4cec7_calibrate_1039770_6.stderr parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d2848c2b10d129e6f94b89071888144 with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7ae14e33-a555-4d50-a896-8d0453d5a612_calibrate_1039769_47.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/7ae14e33-a555-4d50-a896-8d0453d5a612_calibrate_1039769_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cac254fbb8dc5c6784b96e25e03b20c5 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/122e7af4-d3c0-486f-a487-63e814ee6c8d_calibrate_1039768_11.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/122e7af4-d3c0-486f-a487-63e814ee6c8d_calibrate_1039768_11.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4752ddda3c70b9e28e953fbf8b5cc45d with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3484d90f-346e-41e4-940d-4f4cdcaeca09_calibrate_1039770_47.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/3484d90f-346e-41e4-940d-4f4cdcaeca09_calibrate_1039770_47.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2741e0afe8ecbe8894b34da11c6510fd with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fc2b5bab-1ac3-47fa-acb0-71db21e93ee8_calibrate_1039770_54.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/fc2b5bab-1ac3-47fa-acb0-71db21e93ee8_calibrate_1039770_54.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee270d1ca0eaf01c2978b9f39eecabae with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fe6320c4-bb23-4f3c-af7b-db0e30da08eb_calibrate_1039768_29.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/fe6320c4-bb23-4f3c-af7b-db0e30da08eb_calibrate_1039768_29.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9dd9dd9d884602602817bc7aa4f3aa1 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/1959221c-d848-47df-b5ee-da3581b8fc91_calibrate_1039768_45.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/1959221c-d848-47df-b5ee-da3581b8fc91_calibrate_1039768_45.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f343f77efcbf231f4f83a39f1acb6d with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/bb57892e-4e7a-45fe-a08b-89fc3f57b7cd_calibrate_1039770_38.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/bb57892e-4e7a-45fe-a08b-89fc3f57b7cd_calibrate_1039770_38.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 34 parsl.dataflow.strategy DEBUG: Executor multi has 34 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86d030a57eedba7fe8a24789cfffbc14 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/66d3a77f-4f85-4cbe-9e16-b70b4a65e287_calibrate_1039769_58.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/66d3a77f-4f85-4cbe-9e16-b70b4a65e287_calibrate_1039769_58.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a61d2707d66958ab3aa940ef68acac1 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/07431680-1b20-4355-85f8-a90c637c1922_calibrate_1039769_53.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/07431680-1b20-4355-85f8-a90c637c1922_calibrate_1039769_53.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adc269a97b0a306dc889aec65f260203 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/70df13f1-da17-4f3f-bb18-fd85072221a2_calibrate_1039769_32.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/70df13f1-da17-4f3f-bb18-fd85072221a2_calibrate_1039769_32.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b79980f828e3e0b33807b2bb9621f5e6 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec6c1845-bd6a-4875-94e4-98ec8966b38a_calibrate_1039768_52.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/ec6c1845-bd6a-4875-94e4-98ec8966b38a_calibrate_1039768_52.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db0af27e58d2546c7895e52ec198d48 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/eb45920c-ca53-4457-adb4-275e62ac51c2_calibrate_1039768_34.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/eb45920c-ca53-4457-adb4-275e62ac51c2_calibrate_1039768_34.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8d4ff1e38d2b69b89e217357462b220 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash e02f71a713a290502f7db510bcac2647 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e32e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f39459fc-2a8b-40f8-a7d4-33f9e28fbe78_calibrate_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f39459fc-2a8b-40f8-a7d4-33f9e28fbe78_calibrate_1039770_9.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ccc570-0da4-41d4-926a-e97ccdde5960_characterizeImage_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039770/c9ccc570-0da4-41d4-926a-e97ccdde5960_characterizeImage_1039770_9.stderr parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry befd76c4245e2d3698a1acffa5638c89 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/82db887a-b2b2-4e9a-8ba3-c3f15d7b6945_calibrate_1039770_12.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/82db887a-b2b2-4e9a-8ba3-c3f15d7b6945_calibrate_1039770_12.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e3f844a389ea545947ad4b5b0d7a71b with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/77fde3fe-e3bd-4835-a6d0-d413195c7105_calibrate_1039768_23.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/77fde3fe-e3bd-4835-a6d0-d413195c7105_calibrate_1039768_23.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8487576615c274008bd36074b0007bd with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d8d4adab-4b41-4890-8225-a29ae65bc984_calibrate_1039768_19.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/d8d4adab-4b41-4890-8225-a29ae65bc984_calibrate_1039768_19.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0582f515599f78ed320eb694195b0dc2 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/da483fe9-1fea-458f-bf73-17a700c7680b_calibrate_1039672_24.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/da483fe9-1fea-458f-bf73-17a700c7680b_calibrate_1039672_24.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4454d67cc9fd962031c9fc76eb50eba with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/44ebc5fc-0733-4540-8f58-0268fa2bf1f3_calibrate_1039672_47.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/44ebc5fc-0733-4540-8f58-0268fa2bf1f3_calibrate_1039672_47.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 178ac80a2a531961b4d358cde849393e with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/64863aee-88f0-44ae-972e-1b617a37d931_calibrate_1039770_15.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/64863aee-88f0-44ae-972e-1b617a37d931_calibrate_1039770_15.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe5c3e69a209184eba20efbcbd6c014 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5e82b926-81a9-447f-b3ba-c5680b0ee621_calibrate_1039769_15.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/5e82b926-81a9-447f-b3ba-c5680b0ee621_calibrate_1039769_15.stderr 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5ecd4db13fbc6d58bc9632137cf867a with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/73e3d084-7e87-4ac6-8a4b-4f6929a4cc35_calibrate_1039673_15.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/73e3d084-7e87-4ac6-8a4b-4f6929a4cc35_calibrate_1039673_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:17 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fd7942ec9832917b598402e332ea5bca with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/53da13c7-cd3c-4cbc-bd6d-cc2e45312746_calibrate_1039769_44.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/53da13c7-cd3c-4cbc-bd6d-cc2e45312746_calibrate_1039769_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:18 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c17125001cc6eed268c8d21b0f94d58 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bd3d5656-adfb-4820-af8e-6f578443fa5f_calibrate_1039768_26.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/bd3d5656-adfb-4820-af8e-6f578443fa5f_calibrate_1039768_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:19 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d7611c85c7647168aa753efff011db1 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/99949db4-49e9-49e5-8c9d-bfd27fb04c4c_calibrate_1039769_21.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/99949db4-49e9-49e5-8c9d-bfd27fb04c4c_calibrate_1039769_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc766b12a68935d9565ebbf6aa0a3332 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8b610b26-c488-4549-a1d1-446bbb85143a_calibrate_1039770_5.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/8b610b26-c488-4549-a1d1-446bbb85143a_calibrate_1039770_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:20 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:21 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eec15d997f5cbd03f7e58242768ab54f with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/86f22a33-6f1a-4f86-ad05-9da3ed340920_calibrate_1039770_59.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/86f22a33-6f1a-4f86-ad05-9da3ed340920_calibrate_1039770_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:22 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:23 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:24 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:25 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4674afa9fd445b5a6670971e9deccd2e with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/79c1a4c7-f7eb-4931-b3f3-c0f57da9e902_calibrate_1039768_9.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/79c1a4c7-f7eb-4931-b3f3-c0f57da9e902_calibrate_1039768_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:26 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1752d64ac5d1e18a7af07ea3d8083ae with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c7c914cb-1fad-4773-af11-3cbd7384bd0b_calibrate_1039770_44.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/c7c914cb-1fad-4773-af11-3cbd7384bd0b_calibrate_1039770_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:27 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f76d07a2e35cf67bc7d0eb21be5ffac1 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4516a2de-5564-4369-bb86-8245ed97130b_calibrate_1039769_9.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/4516a2de-5564-4369-bb86-8245ed97130b_calibrate_1039769_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:28 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31ba8df40e5d4ecde061e90a7efc8483 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b473e5ba-7bd0-4671-9c7b-b85ccbe3cff0_calibrate_1039769_62.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/b473e5ba-7bd0-4671-9c7b-b85ccbe3cff0_calibrate_1039769_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:29 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 668 try 0 failed parsl.dataflow.dflow INFO: Task 668 marked for retry parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 4cc919eda1286cb3083d39753b8c4ea2 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2787c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 1 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1076 try 0 failed parsl.dataflow.dflow INFO: Task 1076 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash ed3c314f097cd3ca1c26ac7bc36f27a8 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e216fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 1 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stderr 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a416b4087d87aebe7322ae0c6410739 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 81ecd55d3c979599435fda21d6007de6 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/524ce03c-fe72-48b6-ad56-dada9b1cfb5f_characterizeImage_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039672/524ce03c-fe72-48b6-ad56-dada9b1cfb5f_characterizeImage_1039672_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:50 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:51 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:52 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:53 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc21e57ce6aad927404d3213fa87f248 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/92f1478d-7f5c-458c-9ce7-0458006e1238_calibrate_1039768_35.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/92f1478d-7f5c-458c-9ce7-0458006e1238_calibrate_1039768_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:54 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:55 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:56 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:57 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:58 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:43:59 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:00 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:01 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:02 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 11 parsl.dataflow.strategy DEBUG: Executor multi has 11 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af1da446146f9bd3dce47a9f0481ef8 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2036e7bd-29a3-4a1a-bd76-47299914de55_calibrate_1039770_62.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/2036e7bd-29a3-4a1a-bd76-47299914de55_calibrate_1039770_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3dde4c58aacf6f96d9cbb8bb0f02ae4e with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 87b1f4ce7adcb81263e0cca204deefe7 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8ab9f918-e3b3-451a-928b-2abd5fad9fe3_characterizeImage_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/8ab9f918-e3b3-451a-928b-2abd5fad9fe3_characterizeImage_1039768_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 10 parsl.dataflow.strategy DEBUG: Executor multi has 10 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1055 try 0 failed parsl.dataflow.dflow INFO: Task 1055 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash 81ecd55d3c979599435fda21d6007de6 parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e215c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 1 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stderr 2024-03-18 04:44:22 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:44:22 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:23 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:24 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:25 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:26 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:27 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e02f71a713a290502f7db510bcac2647 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f39459fc-2a8b-40f8-a7d4-33f9e28fbe78_calibrate_1039770_9.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/f39459fc-2a8b-40f8-a7d4-33f9e28fbe78_calibrate_1039770_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:28 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:29 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:30 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:31 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:32 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:33 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:34 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:35 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:35 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:35 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1076 try 1 failed parsl.dataflow.dflow ERROR: Task 1076 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ed3c314f097cd3ca1c26ac7bc36f27a8 with result from task 1076 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039770/59d936a0-9ea5-4506-afe9-dff9b6864e2d_calibrate_1039770_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 668 try 1 failed parsl.dataflow.dflow ERROR: Task 668 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 4cc919eda1286cb3083d39753b8c4ea2 with result from task 668 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/d861128b-e5dd-4d58-a0dc-079bc2071851_calibrate_1039769_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1055 try 1 failed parsl.dataflow.dflow ERROR: Task 1055 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 81ecd55d3c979599435fda21d6007de6 with result from task 1055 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039672/6a7b0ef3-204a-4863-b207-93b12b67e5e6_calibrate_1039672_61.stderr 2024-03-18 04:44:55 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:44:55 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:55 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:55 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:55 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:56 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:57 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:58 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:44:59 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:00 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:01 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:02 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:03 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:04 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 245 try 0 failed parsl.dataflow.dflow INFO: Task 245 marked for retry parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 87b1f4ce7adcb81263e0cca204deefe7 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f938b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 1 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:05 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:06 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:07 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493d94166c5d5aaa8a7bab47bb73dfa9 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash f932bb5acb0764677f9dd06fd3323811 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91f8d2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/318d650c-b904-4369-8bbc-1c14c11b6391_calibrate_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/318d650c-b904-4369-8bbc-1c14c11b6391_calibrate_1039768_42.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cabc34b9-c051-4b64-bb99-9ba3768d35dc_characterizeImage_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/cabc34b9-c051-4b64-bb99-9ba3768d35dc_characterizeImage_1039768_42.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:28 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:29 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:30 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:31 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:32 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:33 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:34 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10764c51c1ebc1a9d0c76678dfe94b30 with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash b9d2b431ed552b2f37061771de36b4c7 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e2c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 1117 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cdd89880-d8fa-474e-9650-3e207006b6a9_calibrate_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cdd89880-d8fa-474e-9650-3e207006b6a9_calibrate_1039769_7.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/46cf6e8b-04d3-4a36-9d20-020a4085406f_characterizeImage_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039769/46cf6e8b-04d3-4a36-9d20-020a4085406f_characterizeImage_1039769_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:35 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:36 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:37 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:38 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:39 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:40 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:45:41 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:01 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:02 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:03 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:04 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:05 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:06 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:07 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:08 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:09 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:10 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:11 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:12 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:13 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f932bb5acb0764677f9dd06fd3323811 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/318d650c-b904-4369-8bbc-1c14c11b6391_calibrate_1039768_42.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/318d650c-b904-4369-8bbc-1c14c11b6391_calibrate_1039768_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 245 try 1 failed parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.dflow ERROR: Task 245 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.memoization DEBUG: Storing app cache entry 87b1f4ce7adcb81263e0cca204deefe7 with result from task 245 parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/6f71c3dc-5d68-4052-875c-c9b4bc47bf9f_calibrate_1039768_61.stderr parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:46:33 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:34 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51856b382adc62bac861247eec60aacc with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 18cb15cf007eace69d2207ec00baf449 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1118 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/54ad5496-cf2b-451d-8940-c87aa5142c20_characterizeImage_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039674/54ad5496-cf2b-451d-8940-c87aa5142c20_characterizeImage_1039674_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:35 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:36 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:37 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d2b431ed552b2f37061771de36b4c7 with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cdd89880-d8fa-474e-9650-3e207006b6a9_calibrate_1039769_7.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039769/cdd89880-d8fa-474e-9650-3e207006b6a9_calibrate_1039769_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:38 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:39 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:40 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:41 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:42 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:43 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:44 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:45 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:46:46 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 545 try 0 failed parsl.dataflow.dflow INFO: Task 545 marked for retry parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 18cb15cf007eace69d2207ec00baf449 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3bb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 1 launched on executor multi with executor id 1119 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:07 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:08 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:09 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:10 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:11 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:12 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:12 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:12 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:14 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:15 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:16 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:17 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:18 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:19 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:20 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:21 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:21 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:21 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 202c1b3b89a1224cc165b9739013de15 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash e1a068e3ff902ce5e9af4186b8665800 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 1120 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6a7a2f4c-5cd1-4ef9-9ec6-d780821a1d6f_characterizeImage_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039673/6a7a2f4c-5cd1-4ef9-9ec6-d780821a1d6f_characterizeImage_1039673_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:47:41 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:47:41 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 545 try 1 failed parsl.dataflow.dflow ERROR: Task 545 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 18cb15cf007eace69d2207ec00baf449 with result from task 545 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039674/300c6c4e-4bf5-454e-935d-a1fb788810a3_calibrate_1039674_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:41 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:41 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:42 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:43 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:45 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:45 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:46 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:47 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:48 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:49 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:50 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:51 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:52 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:53 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:54 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:47:55 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 572 try 0 failed parsl.dataflow.dflow INFO: Task 572 marked for retry parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash e1a068e3ff902ce5e9af4186b8665800 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3e13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 1 launched on executor multi with executor id 1121 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stderr 2024-03-18 04:48:16 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:48:16 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:17 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:17 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:17 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:18 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:19 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:20 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:21 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:22 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:23 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:24 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:25 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:26 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:27 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:28 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:29 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:30 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 122ae793551358dba8e21ee60f330ee0 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 703e40c6ae2c770019add90f6213e8c8 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14c91e3b91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 1122 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9cdb929e-1c86-4634-9463-376f9a903ff8_calibrate_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9cdb929e-1c86-4634-9463-376f9a903ff8_calibrate_1039768_6.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e3a9db21-6d65-4bd6-9693-8e4df45f034c_characterizeImage_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/characterizeImage/1039768/e3a9db21-6d65-4bd6-9693-8e4df45f034c_characterizeImage_1039768_6.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:50 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:51 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:52 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:53 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:54 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 572 try 1 failed parsl.dataflow.dflow ERROR: Task 572 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e1a068e3ff902ce5e9af4186b8665800 with result from task 572 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039673/26e1a783-45d2-4d43-992e-3eee7ef43a7f_calibrate_1039673_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:55 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:56 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:57 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:58 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:48:59 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:00 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:01 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:02 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 2024-03-18 04:49:22 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:23 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:24 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:25 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 703e40c6ae2c770019add90f6213e8c8 with result from task 494 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1098 parsl.dataflow.dflow INFO: Tasks in state States.failed: 12 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 6 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9cdb929e-1c86-4634-9463-376f9a903ff8_calibrate_1039768_6.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/submit/DEEP/20211002/B1i/science#step1/20240318T043624Z/logs/calibrate/1039768/9cdb929e-1c86-4634-9463-376f9a903ff8_calibrate_1039768_6.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:26 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193637,17193638,17193639,17193640,17193641,17193642,17193643,17193644,17193645,17193646,17193647,17193648,17193649,17193650,17193651,17193652,17193653,17193654,17193655,17193656,17193657,17193658,17193659,17193660,17193661,17193662,17193663,17193664,17193665,17193666' parsl.providers.slurm.slurm DEBUG: squeue returned 17193665 R 17193666 R 17193655 R 17193656 R 17193657 R 17193658 R 17193659 R 17193660 R 17193661 R 17193662 R 17193663 R 17193664 R 17193649 R 17193650 R 17193651 R 17193652 R 17193653 R 17193654 R 17193643 R 17193644 R 17193645 R 17193646 R 17193647 R 17193648 R 17193640 R 17193641 R 17193642 R 17193637 R 17193638 R 17193639 R parsl.providers.slurm.slurm DEBUG: Updating job 17193665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193649 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193641 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193637 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193639 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193667,17193668,17193669,17193670,17193671' parsl.providers.slurm.slurm DEBUG: squeue returned 17193671 R 17193667 R 17193668 R 17193669 R 17193670 R parsl.providers.slurm.slurm DEBUG: Updating job 17193671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193670 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:27 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:28 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:29 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193672,17193673,17193674,17193675,17193676,17193677,17193678,17193680,17193681,17193682,17193683,17193684,17193685,17193686,17193687,17193688,17193689,17193690,17193691,17193692' parsl.providers.slurm.slurm DEBUG: squeue returned 17193690 R 17193691 R 17193692 R 17193683 R 17193684 R 17193685 R 17193686 R 17193687 R 17193688 R 17193689 R 17193672 R 17193673 R 17193674 R 17193675 R 17193676 R 17193677 R 17193678 R 17193680 R 17193681 R 17193682 R parsl.providers.slurm.slurm DEBUG: Updating job 17193690 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193691 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193692 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193684 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193685 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193686 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193687 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193673 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193680 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193681 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193682 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:30 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:31 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:32 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:33 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:34 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17193693,17193694,17193695,17193696,17193697,17193698,17193699,17193700,17193701,17193702,17193703,17193704,17193705,17193706,17193707,17193708,17193709,17193710,17193711,17193712,17193713,17193714,17193715,17193716,17193717,17193718,17193719,17193720,17193721,17193722,17193723,17193724,17193725,17193726,17193727,17193728,17193729' parsl.providers.slurm.slurm DEBUG: squeue returned 17193693 R 17193694 R 17193695 R 17193696 R 17193697 R 17193698 R 17193699 R 17193700 R 17193701 R 17193702 R 17193703 R 17193704 R 17193705 R 17193706 R 17193707 R 17193708 R 17193709 R 17193710 R 17193711 R 17193712 R 17193713 R 17193714 R 17193715 R 17193716 R 17193717 R 17193718 R 17193719 R 17193720 R 17193721 R 17193722 R 17193723 R 17193724 R 17193725 R 17193726 R 17193727 R 17193728 R 17193729 R parsl.providers.slurm.slurm DEBUG: Updating job 17193693 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193694 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193695 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193696 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193697 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193698 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193699 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193700 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193701 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193702 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193703 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193704 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193705 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193706 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193707 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193708 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193709 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193710 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193711 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193712 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193713 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193714 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193715 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193716 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193717 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193719 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17193729 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 2.1457672119140625e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22854082014544 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9c8771d4e91 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9c8771d4e91 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: caaa6f822a34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: caaa6f822a34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf2ee4730087 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf2ee4730087 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4b59c31edea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4b59c31edea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b636a508cef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b636a508cef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68174d4b2213 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68174d4b2213 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67c8902452ad parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67c8902452ad parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 809378915f44 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 809378915f44 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c0ddfc30b57 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c0ddfc30b57 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df28b7babfce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df28b7babfce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: abb4994d8046 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: abb4994d8046 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a75f65a68d36 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a75f65a68d36 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 87c1f9519964 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 87c1f9519964 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84b3f16b4866 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84b3f16b4866 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b40f10091939 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b40f10091939 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 79ae43c14df4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 79ae43c14df4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 62c6895638d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 62c6895638d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8dcb554d2d99 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8dcb554d2d99 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22c84805267e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22c84805267e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8994eb540635 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8994eb540635 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e0fc6bbf76d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e0fc6bbf76d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8fb783d2ffcc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8fb783d2ffcc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 03067017d46d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 03067017d46d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7abbf766e1c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7abbf766e1c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2cd8537c65d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2cd8537c65d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a9911a8a438d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a9911a8a438d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8d0ed690cf39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8d0ed690cf39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9a477ac9bda3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9a477ac9bda3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33484f077529 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33484f077529 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0c50dd5040c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0c50dd5040c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 991bf1b9b33b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 991bf1b9b33b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8350a578b09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8350a578b09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3963591c9226 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3963591c9226 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4b4b95531ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4b4b95531ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c8df01e61c1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c8df01e61c1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14f0db8acc3d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14f0db8acc3d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 81bf0ea5cf25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 81bf0ea5cf25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 77325fb2c232 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 77325fb2c232 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2c80783368bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2c80783368bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 48593f56bd95 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 48593f56bd95 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e31a2981010 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e31a2981010 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5c8998b46278 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5c8998b46278 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae956b739d72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae956b739d72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1db73784e5af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1db73784e5af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dbd528d92f46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dbd528d92f46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: eb11a0c7f236 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: eb11a0c7f236 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f51d9fc22344 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f51d9fc22344 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6543eaa2e81e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6543eaa2e81e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b97487db45b1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b97487db45b1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 07be1fa86bf9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 07be1fa86bf9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18966a2a865f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18966a2a865f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2651f154bd84 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2651f154bd84 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2063bc39ac42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2063bc39ac42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c41ebd7cd43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c41ebd7cd43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e76efe4c3a1a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e76efe4c3a1a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c7468c01417a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c7468c01417a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae53f666029a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae53f666029a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b3c29465fb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b3c29465fb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff253d7b32a1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff253d7b32a1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ec9e60bbab13 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ec9e60bbab13 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5167530c9050 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5167530c9050 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d37abbce9c05 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d37abbce9c05 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 162442119003 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 162442119003 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 44f0f0b606ab parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 44f0f0b606ab parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 382fe9ee41cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 382fe9ee41cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a4753d044a8b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a4753d044a8b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c44d5cd31607 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c44d5cd31607 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7473bab34fb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7473bab34fb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 330149f7d4af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 330149f7d4af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6645ba70282a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6645ba70282a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 829fd94c85d0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 829fd94c85d0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 494153bbd1df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 494153bbd1df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 265c6c4f5efa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 265c6c4f5efa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8513f08c3899 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8513f08c3899 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e027e5e7d9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e027e5e7d9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 205130fdc694 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 205130fdc694 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 269d76243c47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 269d76243c47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4052698c8daa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4052698c8daa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf840a6b8661 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf840a6b8661 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d7ccfd7d911 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d7ccfd7d911 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 90fd8f0d9070 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 90fd8f0d9070 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09e7e45c832b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09e7e45c832b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a486702fd44 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a486702fd44 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 72955531136a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 72955531136a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a0e3d9bc980 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a0e3d9bc980 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c2365db0a1a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c2365db0a1a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe2af254fc3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe2af254fc3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2f5c31445574 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2f5c31445574 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ceb54ebe5a96 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ceb54ebe5a96 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f42ef7de4447 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f42ef7de4447 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0984cf09697 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0984cf09697 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 380f30872f19 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 380f30872f19 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cad194d9eaf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cad194d9eaf 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 2024-03-18 04:49:35 proc_lsst.multi:201 [INFO] cancelling 17193686 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193686 on provider escience_a40 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro 2024-03-18 04:49:35 proc_lsst.multi:201 [INFO] cancelling 17193651 on provider astro proc_lsst.multi INFO: cancelling 17193651 on provider astro 2024-03-18 04:49:35 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint 2024-03-18 04:49:35 proc_lsst.multi:201 [INFO] cancelling 17193720 on provider checkpoint proc_lsst.multi INFO: cancelling 17193720 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193670 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193670 on provider escience_rtx6k 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193643 on provider astro proc_lsst.multi INFO: cancelling 17193643 on provider astro 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193711 on provider checkpoint proc_lsst.multi INFO: cancelling 17193711 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193663 on provider astro proc_lsst.multi INFO: cancelling 17193663 on provider astro 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193723 on provider checkpoint proc_lsst.multi INFO: cancelling 17193723 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193724 on provider checkpoint proc_lsst.multi INFO: cancelling 17193724 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193716 on provider checkpoint proc_lsst.multi INFO: cancelling 17193716 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193694 on provider checkpoint proc_lsst.multi INFO: cancelling 17193694 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193725 on provider checkpoint proc_lsst.multi INFO: cancelling 17193725 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193697 on provider checkpoint proc_lsst.multi INFO: cancelling 17193697 on provider checkpoint 2024-03-18 04:49:36 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro 2024-03-18 04:49:36 proc_lsst.multi:201 [INFO] cancelling 17193646 on provider astro proc_lsst.multi INFO: cancelling 17193646 on provider astro 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193719 on provider checkpoint proc_lsst.multi INFO: cancelling 17193719 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193710 on provider checkpoint proc_lsst.multi INFO: cancelling 17193710 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193673 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193673 on provider escience_a40 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193703 on provider checkpoint proc_lsst.multi INFO: cancelling 17193703 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193717 on provider checkpoint proc_lsst.multi INFO: cancelling 17193717 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193721 on provider checkpoint proc_lsst.multi INFO: cancelling 17193721 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193676 on provider escience_a40 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193713 on provider checkpoint proc_lsst.multi INFO: cancelling 17193713 on provider checkpoint 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193652 on provider astro proc_lsst.multi INFO: cancelling 17193652 on provider astro 2024-03-18 04:49:37 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 2024-03-18 04:49:37 proc_lsst.multi:201 [INFO] cancelling 17193683 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193683 on provider escience_a40 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193722 on provider checkpoint proc_lsst.multi INFO: cancelling 17193722 on provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193705 on provider checkpoint proc_lsst.multi INFO: cancelling 17193705 on provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193677 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193677 on provider escience_a40 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193640 on provider astro proc_lsst.multi INFO: cancelling 17193640 on provider astro 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193653 on provider astro proc_lsst.multi INFO: cancelling 17193653 on provider astro 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193664 on provider astro proc_lsst.multi INFO: cancelling 17193664 on provider astro 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193671 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193671 on provider escience_rtx6k 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193706 on provider checkpoint proc_lsst.multi INFO: cancelling 17193706 on provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193657 on provider astro proc_lsst.multi INFO: cancelling 17193657 on provider astro 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193693 on provider checkpoint proc_lsst.multi INFO: cancelling 17193693 on provider checkpoint 2024-03-18 04:49:38 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro 2024-03-18 04:49:38 proc_lsst.multi:201 [INFO] cancelling 17193658 on provider astro proc_lsst.multi INFO: cancelling 17193658 on provider astro 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193727 on provider checkpoint proc_lsst.multi INFO: cancelling 17193727 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193684 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193684 on provider escience_a40 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 78108 on provider local proc_lsst.multi INFO: cancelling 78108 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 78108 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193718 on provider checkpoint proc_lsst.multi INFO: cancelling 17193718 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193700 on provider checkpoint proc_lsst.multi INFO: cancelling 17193700 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193650 on provider astro proc_lsst.multi INFO: cancelling 17193650 on provider astro 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193698 on provider checkpoint proc_lsst.multi INFO: cancelling 17193698 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193704 on provider checkpoint proc_lsst.multi INFO: cancelling 17193704 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193707 on provider checkpoint proc_lsst.multi INFO: cancelling 17193707 on provider checkpoint 2024-03-18 04:49:39 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 2024-03-18 04:49:39 proc_lsst.multi:201 [INFO] cancelling 17193678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193678 on provider escience_a40 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193729 on provider checkpoint proc_lsst.multi INFO: cancelling 17193729 on provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193726 on provider checkpoint proc_lsst.multi INFO: cancelling 17193726 on provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193714 on provider checkpoint proc_lsst.multi INFO: cancelling 17193714 on provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193688 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193688 on provider escience_a40 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193695 on provider checkpoint proc_lsst.multi INFO: cancelling 17193695 on provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193642 on provider astro proc_lsst.multi INFO: cancelling 17193642 on provider astro 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193696 on provider checkpoint proc_lsst.multi INFO: cancelling 17193696 on provider checkpoint 2024-03-18 04:49:40 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 2024-03-18 04:49:40 proc_lsst.multi:201 [INFO] cancelling 17193680 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193680 on provider escience_a40 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193715 on provider checkpoint proc_lsst.multi INFO: cancelling 17193715 on provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193649 on provider astro proc_lsst.multi INFO: cancelling 17193649 on provider astro 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193639 on provider astro proc_lsst.multi INFO: cancelling 17193639 on provider astro 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193661 on provider astro proc_lsst.multi INFO: cancelling 17193661 on provider astro 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193654 on provider astro proc_lsst.multi INFO: cancelling 17193654 on provider astro 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193699 on provider checkpoint proc_lsst.multi INFO: cancelling 17193699 on provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193728 on provider checkpoint proc_lsst.multi INFO: cancelling 17193728 on provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193702 on provider checkpoint proc_lsst.multi INFO: cancelling 17193702 on provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193709 on provider checkpoint proc_lsst.multi INFO: cancelling 17193709 on provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint 2024-03-18 04:49:41 proc_lsst.multi:201 [INFO] cancelling 17193708 on provider checkpoint proc_lsst.multi INFO: cancelling 17193708 on provider checkpoint 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193672 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193672 on provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193701 on provider checkpoint proc_lsst.multi INFO: cancelling 17193701 on provider checkpoint 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193685 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193685 on provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193637 on provider astro proc_lsst.multi INFO: cancelling 17193637 on provider astro 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193691 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193691 on provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193682 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193682 on provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193668 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193668 on provider escience_rtx6k 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193675 on provider escience_a40 2024-03-18 04:49:42 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro 2024-03-18 04:49:42 proc_lsst.multi:201 [INFO] cancelling 17193662 on provider astro proc_lsst.multi INFO: cancelling 17193662 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193647 on provider astro proc_lsst.multi INFO: cancelling 17193647 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193666 on provider astro proc_lsst.multi INFO: cancelling 17193666 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193712 on provider checkpoint proc_lsst.multi INFO: cancelling 17193712 on provider checkpoint 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193659 on provider astro proc_lsst.multi INFO: cancelling 17193659 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193656 on provider astro proc_lsst.multi INFO: cancelling 17193656 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193674 on provider escience_a40 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193669 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193669 on provider escience_rtx6k 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193648 on provider astro proc_lsst.multi INFO: cancelling 17193648 on provider astro 2024-03-18 04:49:43 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro 2024-03-18 04:49:43 proc_lsst.multi:201 [INFO] cancelling 17193641 on provider astro proc_lsst.multi INFO: cancelling 17193641 on provider astro 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193667 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193667 on provider escience_rtx6k 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193645 on provider astro proc_lsst.multi INFO: cancelling 17193645 on provider astro 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193665 on provider astro proc_lsst.multi INFO: cancelling 17193665 on provider astro 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193660 on provider astro proc_lsst.multi INFO: cancelling 17193660 on provider astro 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193644 on provider astro proc_lsst.multi INFO: cancelling 17193644 on provider astro 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193681 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193681 on provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193692 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193692 on provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193690 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193690 on provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193689 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193689 on provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 2024-03-18 04:49:44 proc_lsst.multi:201 [INFO] cancelling 17193687 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193687 on provider escience_a40 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193638 on provider astro proc_lsst.multi INFO: cancelling 17193638 on provider astro 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193655 on provider astro proc_lsst.multi INFO: cancelling 17193655 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-18 04:49:45 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-18 04:49:45 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17193693', '17193694', '17193695', '17193696', '17193697', '17193698', '17193699', '17193700', '17193701', '17193702', '17193703', '17193704', '17193705', '17193706', '17193707', '17193708', '17193709', '17193710', '17193711', '17193712', '17193713', '17193714', '17193715', '17193716', '17193717', '17193718', '17193719', '17193720', '17193721', '17193722', '17193723', '17193724', '17193725', '17193726', '17193727', '17193728', '17193729', '17193637', '17193638', '17193639', '17193640', '17193641', '17193642', '17193643', '17193644', '17193645', '17193646', '17193647', '17193648', '17193649', '17193650', '17193651', '17193652', '17193653', '17193654', '17193655', '17193656', '17193657', '17193658', '17193659', '17193660', '17193661', '17193662', '17193663', '17193664', '17193665', '17193666', '17193672', '17193673', '17193674', '17193675', '17193676', '17193677', '17193678', '17193680', '17193681', '17193682', '17193683', '17193684', '17193685', '17193686', '17193687', '17193688', '17193689', '17193690', '17193691', '17193692', '17193667', '17193668', '17193669', '17193670', '17193671', '78108'] proc_lsst.multi INFO: new jobs since last cancel ['17193693', '17193694', '17193695', '17193696', '17193697', '17193698', '17193699', '17193700', '17193701', '17193702', '17193703', '17193704', '17193705', '17193706', '17193707', '17193708', '17193709', '17193710', '17193711', '17193712', '17193713', '17193714', '17193715', '17193716', '17193717', '17193718', '17193719', '17193720', '17193721', '17193722', '17193723', '17193724', '17193725', '17193726', '17193727', '17193728', '17193729', '17193637', '17193638', '17193639', '17193640', '17193641', '17193642', '17193643', '17193644', '17193645', '17193646', '17193647', '17193648', '17193649', '17193650', '17193651', '17193652', '17193653', '17193654', '17193655', '17193656', '17193657', '17193658', '17193659', '17193660', '17193661', '17193662', '17193663', '17193664', '17193665', '17193666', '17193672', '17193673', '17193674', '17193675', '17193676', '17193677', '17193678', '17193680', '17193681', '17193682', '17193683', '17193684', '17193685', '17193686', '17193687', '17193688', '17193689', '17193690', '17193691', '17193692', '17193667', '17193668', '17193669', '17193670', '17193671', '78108'] 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193693 in provider checkpoint proc_lsst.multi INFO: found job 17193693 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193693 on provider checkpoint proc_lsst.multi INFO: cancelling 17193693 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193694 in provider checkpoint proc_lsst.multi INFO: found job 17193694 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193694 on provider checkpoint proc_lsst.multi INFO: cancelling 17193694 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193695 in provider checkpoint proc_lsst.multi INFO: found job 17193695 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193695 on provider checkpoint proc_lsst.multi INFO: cancelling 17193695 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193696 in provider checkpoint proc_lsst.multi INFO: found job 17193696 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193696 on provider checkpoint proc_lsst.multi INFO: cancelling 17193696 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193697 in provider checkpoint proc_lsst.multi INFO: found job 17193697 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193697 on provider checkpoint proc_lsst.multi INFO: cancelling 17193697 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193698 in provider checkpoint proc_lsst.multi INFO: found job 17193698 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193698 on provider checkpoint proc_lsst.multi INFO: cancelling 17193698 on provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:146 [INFO] found job 17193699 in provider checkpoint proc_lsst.multi INFO: found job 17193699 in provider checkpoint 2024-03-18 04:49:45 proc_lsst.multi:201 [INFO] cancelling 17193699 on provider checkpoint proc_lsst.multi INFO: cancelling 17193699 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193700 in provider checkpoint proc_lsst.multi INFO: found job 17193700 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193700 on provider checkpoint proc_lsst.multi INFO: cancelling 17193700 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193701 in provider checkpoint proc_lsst.multi INFO: found job 17193701 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193701 on provider checkpoint proc_lsst.multi INFO: cancelling 17193701 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193702 in provider checkpoint proc_lsst.multi INFO: found job 17193702 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193702 on provider checkpoint proc_lsst.multi INFO: cancelling 17193702 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193703 in provider checkpoint proc_lsst.multi INFO: found job 17193703 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193703 on provider checkpoint proc_lsst.multi INFO: cancelling 17193703 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193704 in provider checkpoint proc_lsst.multi INFO: found job 17193704 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193704 on provider checkpoint proc_lsst.multi INFO: cancelling 17193704 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193705 in provider checkpoint proc_lsst.multi INFO: found job 17193705 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193705 on provider checkpoint proc_lsst.multi INFO: cancelling 17193705 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193706 in provider checkpoint proc_lsst.multi INFO: found job 17193706 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193706 on provider checkpoint proc_lsst.multi INFO: cancelling 17193706 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193707 in provider checkpoint proc_lsst.multi INFO: found job 17193707 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193707 on provider checkpoint proc_lsst.multi INFO: cancelling 17193707 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193708 in provider checkpoint proc_lsst.multi INFO: found job 17193708 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193708 on provider checkpoint proc_lsst.multi INFO: cancelling 17193708 on provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:146 [INFO] found job 17193709 in provider checkpoint proc_lsst.multi INFO: found job 17193709 in provider checkpoint 2024-03-18 04:49:46 proc_lsst.multi:201 [INFO] cancelling 17193709 on provider checkpoint proc_lsst.multi INFO: cancelling 17193709 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193710 in provider checkpoint proc_lsst.multi INFO: found job 17193710 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193710 on provider checkpoint proc_lsst.multi INFO: cancelling 17193710 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193711 in provider checkpoint proc_lsst.multi INFO: found job 17193711 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193711 on provider checkpoint proc_lsst.multi INFO: cancelling 17193711 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193712 in provider checkpoint proc_lsst.multi INFO: found job 17193712 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193712 on provider checkpoint proc_lsst.multi INFO: cancelling 17193712 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193713 in provider checkpoint proc_lsst.multi INFO: found job 17193713 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193713 on provider checkpoint proc_lsst.multi INFO: cancelling 17193713 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193714 in provider checkpoint proc_lsst.multi INFO: found job 17193714 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193714 on provider checkpoint proc_lsst.multi INFO: cancelling 17193714 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193715 in provider checkpoint proc_lsst.multi INFO: found job 17193715 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193715 on provider checkpoint proc_lsst.multi INFO: cancelling 17193715 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193716 in provider checkpoint proc_lsst.multi INFO: found job 17193716 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193716 on provider checkpoint proc_lsst.multi INFO: cancelling 17193716 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193717 in provider checkpoint proc_lsst.multi INFO: found job 17193717 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193717 on provider checkpoint proc_lsst.multi INFO: cancelling 17193717 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193718 in provider checkpoint proc_lsst.multi INFO: found job 17193718 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193718 on provider checkpoint proc_lsst.multi INFO: cancelling 17193718 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193719 in provider checkpoint proc_lsst.multi INFO: found job 17193719 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193719 on provider checkpoint proc_lsst.multi INFO: cancelling 17193719 on provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:146 [INFO] found job 17193720 in provider checkpoint proc_lsst.multi INFO: found job 17193720 in provider checkpoint 2024-03-18 04:49:47 proc_lsst.multi:201 [INFO] cancelling 17193720 on provider checkpoint proc_lsst.multi INFO: cancelling 17193720 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193721 in provider checkpoint proc_lsst.multi INFO: found job 17193721 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193721 on provider checkpoint proc_lsst.multi INFO: cancelling 17193721 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193722 in provider checkpoint proc_lsst.multi INFO: found job 17193722 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193722 on provider checkpoint proc_lsst.multi INFO: cancelling 17193722 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193723 in provider checkpoint proc_lsst.multi INFO: found job 17193723 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193723 on provider checkpoint proc_lsst.multi INFO: cancelling 17193723 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193724 in provider checkpoint proc_lsst.multi INFO: found job 17193724 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193724 on provider checkpoint proc_lsst.multi INFO: cancelling 17193724 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193725 in provider checkpoint proc_lsst.multi INFO: found job 17193725 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193725 on provider checkpoint proc_lsst.multi INFO: cancelling 17193725 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193726 in provider checkpoint proc_lsst.multi INFO: found job 17193726 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193726 on provider checkpoint proc_lsst.multi INFO: cancelling 17193726 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193727 in provider checkpoint proc_lsst.multi INFO: found job 17193727 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193727 on provider checkpoint proc_lsst.multi INFO: cancelling 17193727 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193728 in provider checkpoint proc_lsst.multi INFO: found job 17193728 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193728 on provider checkpoint proc_lsst.multi INFO: cancelling 17193728 on provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:146 [INFO] found job 17193729 in provider checkpoint proc_lsst.multi INFO: found job 17193729 in provider checkpoint 2024-03-18 04:49:48 proc_lsst.multi:201 [INFO] cancelling 17193729 on provider checkpoint proc_lsst.multi INFO: cancelling 17193729 on provider checkpoint 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193637 in provider astro proc_lsst.multi INFO: found job 17193637 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193637 on provider astro proc_lsst.multi INFO: cancelling 17193637 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193638 in provider astro proc_lsst.multi INFO: found job 17193638 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193638 on provider astro proc_lsst.multi INFO: cancelling 17193638 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193639 in provider astro proc_lsst.multi INFO: found job 17193639 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193639 on provider astro proc_lsst.multi INFO: cancelling 17193639 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193640 in provider astro proc_lsst.multi INFO: found job 17193640 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193640 on provider astro proc_lsst.multi INFO: cancelling 17193640 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193641 in provider astro proc_lsst.multi INFO: found job 17193641 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193641 on provider astro proc_lsst.multi INFO: cancelling 17193641 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193642 in provider astro proc_lsst.multi INFO: found job 17193642 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193642 on provider astro proc_lsst.multi INFO: cancelling 17193642 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193643 in provider astro proc_lsst.multi INFO: found job 17193643 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193643 on provider astro proc_lsst.multi INFO: cancelling 17193643 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193644 in provider astro proc_lsst.multi INFO: found job 17193644 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193644 on provider astro proc_lsst.multi INFO: cancelling 17193644 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193645 in provider astro proc_lsst.multi INFO: found job 17193645 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193645 on provider astro proc_lsst.multi INFO: cancelling 17193645 on provider astro 2024-03-18 04:49:49 proc_lsst.multi:146 [INFO] found job 17193646 in provider astro proc_lsst.multi INFO: found job 17193646 in provider astro 2024-03-18 04:49:49 proc_lsst.multi:201 [INFO] cancelling 17193646 on provider astro proc_lsst.multi INFO: cancelling 17193646 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193647 in provider astro proc_lsst.multi INFO: found job 17193647 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193647 on provider astro proc_lsst.multi INFO: cancelling 17193647 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193648 in provider astro proc_lsst.multi INFO: found job 17193648 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193648 on provider astro proc_lsst.multi INFO: cancelling 17193648 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193649 in provider astro proc_lsst.multi INFO: found job 17193649 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193649 on provider astro proc_lsst.multi INFO: cancelling 17193649 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193650 in provider astro proc_lsst.multi INFO: found job 17193650 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193650 on provider astro proc_lsst.multi INFO: cancelling 17193650 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193651 in provider astro proc_lsst.multi INFO: found job 17193651 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193651 on provider astro proc_lsst.multi INFO: cancelling 17193651 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193652 in provider astro proc_lsst.multi INFO: found job 17193652 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193652 on provider astro proc_lsst.multi INFO: cancelling 17193652 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193653 in provider astro proc_lsst.multi INFO: found job 17193653 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193653 on provider astro proc_lsst.multi INFO: cancelling 17193653 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193654 in provider astro proc_lsst.multi INFO: found job 17193654 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193654 on provider astro proc_lsst.multi INFO: cancelling 17193654 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193655 in provider astro proc_lsst.multi INFO: found job 17193655 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193655 on provider astro proc_lsst.multi INFO: cancelling 17193655 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193656 in provider astro proc_lsst.multi INFO: found job 17193656 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193656 on provider astro proc_lsst.multi INFO: cancelling 17193656 on provider astro 2024-03-18 04:49:50 proc_lsst.multi:146 [INFO] found job 17193657 in provider astro proc_lsst.multi INFO: found job 17193657 in provider astro 2024-03-18 04:49:50 proc_lsst.multi:201 [INFO] cancelling 17193657 on provider astro proc_lsst.multi INFO: cancelling 17193657 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193658 in provider astro proc_lsst.multi INFO: found job 17193658 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193658 on provider astro proc_lsst.multi INFO: cancelling 17193658 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193659 in provider astro proc_lsst.multi INFO: found job 17193659 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193659 on provider astro proc_lsst.multi INFO: cancelling 17193659 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193660 in provider astro proc_lsst.multi INFO: found job 17193660 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193660 on provider astro proc_lsst.multi INFO: cancelling 17193660 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193661 in provider astro proc_lsst.multi INFO: found job 17193661 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193661 on provider astro proc_lsst.multi INFO: cancelling 17193661 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193662 in provider astro proc_lsst.multi INFO: found job 17193662 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193662 on provider astro proc_lsst.multi INFO: cancelling 17193662 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193663 in provider astro proc_lsst.multi INFO: found job 17193663 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193663 on provider astro proc_lsst.multi INFO: cancelling 17193663 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193664 in provider astro proc_lsst.multi INFO: found job 17193664 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193664 on provider astro proc_lsst.multi INFO: cancelling 17193664 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193665 in provider astro proc_lsst.multi INFO: found job 17193665 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193665 on provider astro proc_lsst.multi INFO: cancelling 17193665 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193666 in provider astro proc_lsst.multi INFO: found job 17193666 in provider astro 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193666 on provider astro proc_lsst.multi INFO: cancelling 17193666 on provider astro 2024-03-18 04:49:51 proc_lsst.multi:146 [INFO] found job 17193672 in provider escience_a40 proc_lsst.multi INFO: found job 17193672 in provider escience_a40 2024-03-18 04:49:51 proc_lsst.multi:201 [INFO] cancelling 17193672 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193672 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193673 in provider escience_a40 proc_lsst.multi INFO: found job 17193673 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193673 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193673 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193674 in provider escience_a40 proc_lsst.multi INFO: found job 17193674 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193674 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193675 in provider escience_a40 proc_lsst.multi INFO: found job 17193675 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193675 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193676 in provider escience_a40 proc_lsst.multi INFO: found job 17193676 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193676 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193677 in provider escience_a40 proc_lsst.multi INFO: found job 17193677 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193677 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193677 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193678 in provider escience_a40 proc_lsst.multi INFO: found job 17193678 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193678 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193680 in provider escience_a40 proc_lsst.multi INFO: found job 17193680 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193680 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193680 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193681 in provider escience_a40 proc_lsst.multi INFO: found job 17193681 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193681 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193681 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193682 in provider escience_a40 proc_lsst.multi INFO: found job 17193682 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193682 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193682 on provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:146 [INFO] found job 17193683 in provider escience_a40 proc_lsst.multi INFO: found job 17193683 in provider escience_a40 2024-03-18 04:49:52 proc_lsst.multi:201 [INFO] cancelling 17193683 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193683 on provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:146 [INFO] found job 17193684 in provider escience_a40 proc_lsst.multi INFO: found job 17193684 in provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:201 [INFO] cancelling 17193684 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193684 on provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:146 [INFO] found job 17193685 in provider escience_a40 proc_lsst.multi INFO: found job 17193685 in provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:201 [INFO] cancelling 17193685 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193685 on provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:146 [INFO] found job 17193686 in provider escience_a40 proc_lsst.multi INFO: found job 17193686 in provider escience_a40 2024-03-18 04:49:53 proc_lsst.multi:201 [INFO] cancelling 17193686 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193686 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193687 in provider escience_a40 proc_lsst.multi INFO: found job 17193687 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193687 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193687 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193688 in provider escience_a40 proc_lsst.multi INFO: found job 17193688 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193688 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193688 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193689 in provider escience_a40 proc_lsst.multi INFO: found job 17193689 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193689 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193689 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193690 in provider escience_a40 proc_lsst.multi INFO: found job 17193690 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193690 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193690 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193691 in provider escience_a40 proc_lsst.multi INFO: found job 17193691 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193691 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193691 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193692 in provider escience_a40 proc_lsst.multi INFO: found job 17193692 in provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193692 on provider escience_a40 proc_lsst.multi INFO: cancelling 17193692 on provider escience_a40 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193667 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193667 in provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193667 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193667 on provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193668 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193668 in provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193668 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193668 on provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193669 in provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193669 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193669 on provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:146 [INFO] found job 17193670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193670 in provider escience_rtx6k 2024-03-18 04:49:54 proc_lsst.multi:201 [INFO] cancelling 17193670 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193670 on provider escience_rtx6k 2024-03-18 04:49:55 proc_lsst.multi:146 [INFO] found job 17193671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17193671 in provider escience_rtx6k 2024-03-18 04:49:55 proc_lsst.multi:201 [INFO] cancelling 17193671 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17193671 on provider escience_rtx6k 2024-03-18 04:49:55 proc_lsst.multi:146 [INFO] found job 78108 in provider local proc_lsst.multi INFO: found job 78108 in provider local 2024-03-18 04:49:55 proc_lsst.multi:201 [INFO] cancelling 78108 on provider local proc_lsst.multi INFO: cancelling 78108 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 78108 parsl.providers.local.local WARNING: Failed to kill PID: 78108 and child processes on local 2024-03-18 04:49:56 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 770.0222 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20211002_B1i_science#step1_20240318T043624Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 770.0287 seconds; current memory usage: 0.386 Gibyte, delta: 0.009 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 808.3248 seconds; current memory usage: 0.386 Gibyte, delta: 0.250 Gibyte, peak delta: 0.250 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.386 Gibyte (main), 0.386 Gibyte (largest child process) Run Id: None Run Name: DEEP_20211002_B1i_science#step1_20240318T043624Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 13m30.706s user 3m0.239s sys 1m46.913s